2023-12-02T01:03:55.952 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-02T01:03:55.952 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} --archive /home/teuthworker/archive/yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi/7474671 --verbose --name yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi/7474671/orig.config.yaml 2023-12-02T01:03:55.975 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-02T01:03:56.045 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi/7474671 branch: main description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7474671' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bed7681480ccb0192b91a35e691658867e1e96b7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bed7681480ccb0192b91a35e691658867e1e96b7 workunit: branch: main sha1: bed7681480ccb0192b91a35e691658867e1e96b7 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1147 sha1: bed7681480ccb0192b91a35e691658867e1e96b7 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bed7681480ccb0192b91a35e691658867e1e96b7 targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPR7i45QAtTsn88VNEd9XXlKC4pSCTBbv/WfCIDn1qnRaWhTGoTDrSoN5HiZ/jk3/ndyrs7Jr0VgLPJWRLqbCSM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-01_17:58:18 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-02T01:03:56.045 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa; will attempt to use it 2023-12-02T01:03:56.046 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks 2023-12-02T01:03:56.046 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-02T01:03:56.048 INFO:teuthology.task.internal:Checking packages... 2023-12-02T01:03:56.071 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bed7681480ccb0192b91a35e691658867e1e96b7' 2023-12-02T01:03:56.071 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-02T01:03:56.071 INFO:teuthology.packaging:ref: None 2023-12-02T01:03:56.071 INFO:teuthology.packaging:tag: None 2023-12-02T01:03:56.071 INFO:teuthology.packaging:branch: main 2023-12-02T01:03:56.071 INFO:teuthology.packaging:sha1: bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T01:03:56.071 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-12-02T01:03:56.291 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7660.gb27938f5 2023-12-02T01:03:56.291 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-02T01:03:56.298 INFO:teuthology.task.internal:no buildpackages task found 2023-12-02T01:03:56.298 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-02T01:03:56.317 INFO:teuthology.task.internal:Saving configuration 2023-12-02T01:03:56.326 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-02T01:03:56.333 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T01:03:56.359 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi/7474671', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 00:58:48.787937', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPR7i45QAtTsn88VNEd9XXlKC4pSCTBbv/WfCIDn1qnRaWhTGoTDrSoN5HiZ/jk3/ndyrs7Jr0VgLPJWRLqbCSM='} 2023-12-02T01:03:56.359 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-02T01:03:56.366 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-12-02T01:03:56.366 INFO:teuthology.run_tasks:Running task console_log... 2023-12-02T01:03:56.406 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6a2ba9dee0>, signals=[15]) 2023-12-02T01:03:56.407 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-02T01:03:56.413 INFO:teuthology.task.internal:Opening connections... 2023-12-02T01:03:56.413 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2023-12-02T01:03:56.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T01:03:56.498 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-02T01:03:56.504 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2023-12-02T01:03:56.567 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2023-12-02T01:03:56.567 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2023-12-02T01:03:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2023-12-02T01:03:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2023-12-02T01:03:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2023-12-02T01:03:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2023-12-02T01:03:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2023-12-02T01:03:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2023-12-02T01:03:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-02T01:03:56.625 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2023-12-02T01:03:56.625 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2023-12-02T01:03:56.625 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-02T01:03:56.625 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2023-12-02T01:03:56.625 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-02T01:03:56.625 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-02T01:03:56.625 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-02T01:03:56.625 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2023-12-02T01:03:56.644 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-02T01:03:56.653 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-02T01:03:56.663 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-02T01:03:56.663 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2023-12-02T01:03:56.681 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-02T01:03:56.686 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-02T01:03:56.687 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2023-12-02T01:03:56.750 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-02T01:03:56.777 INFO:teuthology.run_tasks:Running task kernel... 2023-12-02T01:03:56.793 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-02T01:03:56.793 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-02T01:03:56.794 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-02T01:03:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T01:03:56.812 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2023-12-02T01:03:56.870 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-361.el9.x86_64 2023-12-02T01:03:56.871 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-361.el9.x86_64 2023-12-02T01:03:56.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2023-12-02T01:03:59.093 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-12-02T01:04:00.501 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2023-12-02T01:04:01.417 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-12-02T01:04:03.185 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 32 MB/s | 18 MB 00:00 2023-12-02T01:04:08.237 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.8 MB 00:01 2023-12-02T01:04:09.510 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 92 kB/s | 15 kB 00:00 2023-12-02T01:04:10.654 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 20 MB/s | 20 MB 00:00 2023-12-02T01:04:15.494 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-12-02T01:04:17.826 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-02T01:04:17.826 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-02T01:04:17.945 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:04:17.957 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:04:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2023-12-02T01:04:17.957 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:04:17.957 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:04:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2023-12-02T01:04:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:04:17.959 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 96 M 2023-12-02T01:04:17.959 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 121 M 2023-12-02T01:04:17.959 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:04:20.037 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 3.1 MB/s | 5.6 MB 00:01 2023-12-02T01:04:22.420 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 4.9 MB/s | 20 MB 00:04 2023-12-02T01:04:27.128 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 4.3 MB/s | 38 MB 00:08 2023-12-02T01:04:28.028 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 4.0 MB/s | 32 MB 00:07 2023-12-02T01:04:28.029 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:04:28.029 INFO:teuthology.orchestra.run.smithi162.stdout:Total 9.5 MB/s | 96 MB 00:10 2023-12-02T01:04:28.887 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:04:29.302 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:04:29.302 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:04:30.642 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:04:30.642 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:04:33.278 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:04:33.907 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-02T01:04:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T01:04:36.791 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T01:04:36.970 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T01:04:41.563 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T01:04:41.693 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T01:04:46.309 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T01:05:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T01:05:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-02T01:05:15.714 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T01:05:15.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-02T01:05:15.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T01:05:15.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T01:05:16.131 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T01:05:16.131 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:05:16.132 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:05:16.132 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-02T01:05:16.132 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-02T01:05:16.132 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-02T01:05:16.132 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-02T01:05:16.132 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:05:16.132 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:05:16.601 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T01:05:16.738 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T01:05:16.738 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T01:05:16.738 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-02T01:05:16.738 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-02T01:05:16.738 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-02T01:05:16.739 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-02T01:05:16.739 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2023-12-02T01:05:16.739 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T01:05:16.856 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T01:05:16.856 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T01:05:16.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-02T01:05:16.891 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:05:16.892 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2023-12-02T01:05:16.893 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-02T01:05:16.893 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-388.el9.x86_64 2023-12-02T01:05:16.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-02T01:05:17.542 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2023-12-02T01:05:18.929 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-02T01:05:18.961 INFO:teuthology.orchestra.run.smithi162.stderr:done 2023-12-02T01:05:18.966 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2023-12-02T01:05:18.985 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.EAQ5KFiwdA 2023-12-02T01:05:18.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EAQ5KFiwdA 2023-12-02T01:05:19.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.EAQ5KFiwdA 2023-12-02T01:05:19.197 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.EAQ5KFiwdA is 6KB 2023-12-02T01:05:19.245 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.EAQ5KFiwdA 2023-12-02T01:05:19.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2023-12-02T01:05:19.324 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-02T01:05:19.324 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-02T01:05:19.324 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-02T01:05:19.325 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-02T01:05:19.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-02T01:05:19.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2023-12-02T01:05:49.494 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-02T01:05:49.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-02T01:05:49.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T01:06:23.970 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2023-12-02T01:06:32.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-02T01:06:32.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T01:06:51.365 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2023-12-02T01:07:03.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-02T01:07:03.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T01:07:06.057 DEBUG:teuthology.orchestra.run.smithi162:> true 2023-12-02T01:07:06.477 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-02T01:07:06.477 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-02T01:07:06.477 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2023-12-02T01:07:06.491 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-388.el9.x86_64 2023-12-02T01:07:06.491 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-02T01:07:06.491 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-02T01:07:06.491 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-02T01:07:07.492 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-02T01:07:07.492 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-02T01:07:07.584 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2023-12-02T01:07:07.642 DEBUG:teuthology.parallel:result is None 2023-12-02T01:07:07.643 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-02T01:07:07.650 INFO:teuthology.task.internal:Creating test directory... 2023-12-02T01:07:07.650 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-02T01:07:07.665 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-02T01:07:07.671 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-02T01:07:07.679 INFO:teuthology.task.internal:Creating archive directory... 2023-12-02T01:07:07.680 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-02T01:07:07.733 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-02T01:07:07.741 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-02T01:07:07.741 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-02T01:07:07.818 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:07:07.831 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:07:07.832 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-02T01:07:07.839 INFO:teuthology.task.internal:Configuring sudo... 2023-12-02T01:07:07.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-02T01:07:07.866 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-02T01:07:07.875 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-02T01:07:07.875 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-02T01:07:07.921 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T01:07:08.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T01:07:08.055 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T01:07:08.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T01:07:08.177 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-02T01:07:08.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-02T01:07:08.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2023-12-02T01:07:08.337 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T01:07:08.823 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-02T01:07:08.831 INFO:teuthology.task.internal:Starting timer... 2023-12-02T01:07:08.831 INFO:teuthology.run_tasks:Running task pcp... 2023-12-02T01:07:08.872 INFO:teuthology.run_tasks:Running task selinux... 2023-12-02T01:07:08.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2023-12-02T01:07:08.963 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2023-12-02T01:07:08.965 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-02T01:07:08.965 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2023-12-02T01:07:08.980 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2023-12-02T01:07:08.980 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2023-12-02T01:07:08.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T01:07:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T01:07:09.048 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-02T01:07:09.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2023-12-02T01:07:09.123 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-02T01:07:09.134 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-02T01:07:09.140 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-02T01:07:09.278 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-02T01:07:09.297 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-02T01:07:09.298 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-02T01:12:49.630 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2023-12-02T01:12:49.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-02T01:12:49.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T01:12:49.716 DEBUG:teuthology.orchestra.run.smithi162:> true 2023-12-02T01:12:49.785 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-02T01:12:49.785 INFO:teuthology.run_tasks:Running task clock... 2023-12-02T01:12:49.798 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-02T01:12:49.799 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-02T01:12:49.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T01:12:49.857 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-02T01:12:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-02T01:12:49.903 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2023-12-02T01:12:49.916 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2023-12-02T01:12:49.932 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-02T01:12:49.948 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-02T01:12:49.991 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2023-12-02T01:12:49.995 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T01:12:49.995 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2023-12-02T01:12:49.995 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T01:12:49.995 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T01:12:49.995 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T01:12:49.995 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T01:12:49.995 INFO:teuthology.run_tasks:Running task install... 2023-12-02T01:12:50.006 DEBUG:teuthology.task.install:project ceph 2023-12-02T01:12:50.006 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}} 2023-12-02T01:12:50.006 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'} 2023-12-02T01:12:50.006 INFO:teuthology.task.install:Using flavor: default 2023-12-02T01:12:50.012 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-02T01:12:50.012 INFO:teuthology.task.install:extra packages: [] 2023-12-02T01:12:50.012 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7', 'tag': None, 'wait_for_package': False} 2023-12-02T01:12:50.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T01:12:50.157 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/ 2023-12-02T01:12:50.157 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7631.gbed76814 2023-12-02T01:12:50.264 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-02T01:12:50.264 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-02T01:12:50.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-02T01:12:50.292 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-02T01:12:50.292 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bed7681480ccb0192b91a35e691658867e1e96b7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-02T01:12:50.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-02T01:12:50.440 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-02T01:12:50.507 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2023-12-02T01:12:50.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2023-12-02T01:12:50.750 INFO:teuthology.orchestra.run.smithi162.stdout:73 files removed 2023-12-02T01:12:50.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2023-12-02T01:12:51.605 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 153 kB/s | 78 kB 00:00 2023-12-02T01:12:52.131 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 20 kB/s | 9.5 kB 00:00 2023-12-02T01:12:52.698 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 3.5 kB/s | 1.9 kB 00:00 2023-12-02T01:12:53.434 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 9.1 MB/s | 6.4 MB 00:00 2023-12-02T01:12:54.809 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2023-12-02T01:13:00.349 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 1.5 MB/s | 7.9 MB 00:05 2023-12-02T01:13:06.216 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 18 MB 00:04 2023-12-02T01:13:11.008 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.8 MB 00:01 2023-12-02T01:13:13.235 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 15 kB 00:01 2023-12-02T01:13:19.401 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 3.3 MB/s | 20 MB 00:06 2023-12-02T01:13:24.234 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-12-02T01:13:26.909 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:13:26.910 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:13:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:13:26.910 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:18.0.0-7631.gbed76814.el9 ceph 8.6 M 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.3 M 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.0 M 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:18.0.0-7631.gbed76814.el9 ceph 5.2 M 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:18.0.0-7631.gbed76814.el9 ceph 19 M 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:18.0.0-7631.gbed76814.el9 ceph 24 k 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-02T01:13:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 718 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 475 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 4.8 M 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: x86_64 2:18.0.0-7631.gbed76814.el9 ceph 45 k 2023-12-02T01:13:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:18.0.0-7631.gbed76814.el9 ceph 129 k 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:18.0.0-7631.gbed76814.el9 ceph 162 k 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:18.0.0-7631.gbed76814.el9 ceph 320 k 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 298 k 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:18.0.0-7631.gbed76814.el9 ceph 99 k 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout:Install 27 Packages 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2023-12-02T01:13:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:13:26.914 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 55 M 2023-12-02T01:13:26.914 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:13:29.826 INFO:teuthology.orchestra.run.smithi162.stdout:(1/29): ceph-base-18.0.0-7631.gbed76814.el9.x86 1.8 MB/s | 5.2 MB 00:02 2023-12-02T01:13:29.934 INFO:teuthology.orchestra.run.smithi162.stdout:(2/29): ceph-selinux-18.0.0-7631.gbed76814.el9. 223 kB/s | 24 kB 00:00 2023-12-02T01:13:30.568 INFO:teuthology.orchestra.run.smithi162.stdout:(3/29): libcephfs2-18.0.0-7631.gbed76814.el9.x8 1.1 MB/s | 718 kB 00:00 2023-12-02T01:13:30.727 INFO:teuthology.orchestra.run.smithi162.stdout:(4/29): ceph-radosgw-18.0.0-7631.gbed76814.el9. 2.3 MB/s | 8.6 MB 00:03 2023-12-02T01:13:31.360 INFO:teuthology.orchestra.run.smithi162.stdout:(5/29): libradosstriper1-18.0.0-7631.gbed76814. 601 kB/s | 475 kB 00:00 2023-12-02T01:13:31.444 INFO:teuthology.orchestra.run.smithi162.stdout:(6/29): python3-ceph-argparse-18.0.0-7631.gbed7 542 kB/s | 45 kB 00:00 2023-12-02T01:13:31.536 INFO:teuthology.orchestra.run.smithi162.stdout:(7/29): python3-ceph-common-18.0.0-7631.gbed768 1.4 MB/s | 129 kB 00:00 2023-12-02T01:13:31.629 INFO:teuthology.orchestra.run.smithi162.stdout:(8/29): python3-cephfs-18.0.0-7631.gbed76814.el 1.7 MB/s | 162 kB 00:00 2023-12-02T01:13:31.786 INFO:teuthology.orchestra.run.smithi162.stdout:(9/29): python3-rados-18.0.0-7631.gbed76814.el9 2.0 MB/s | 320 kB 00:00 2023-12-02T01:13:31.912 INFO:teuthology.orchestra.run.smithi162.stdout:(10/29): python3-rbd-18.0.0-7631.gbed76814.el9. 2.3 MB/s | 298 kB 00:00 2023-12-02T01:13:31.985 INFO:teuthology.orchestra.run.smithi162.stdout:(11/29): python3-rgw-18.0.0-7631.gbed76814.el9. 1.3 MB/s | 99 kB 00:00 2023-12-02T01:13:32.169 INFO:teuthology.orchestra.run.smithi162.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 578 kB/s | 106 kB 00:00 2023-12-02T01:13:32.236 INFO:teuthology.orchestra.run.smithi162.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2023-12-02T01:13:32.286 INFO:teuthology.orchestra.run.smithi162.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2023-12-02T01:13:32.403 INFO:teuthology.orchestra.run.smithi162.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2023-12-02T01:13:32.462 INFO:teuthology.orchestra.run.smithi162.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2023-12-02T01:13:32.512 INFO:teuthology.orchestra.run.smithi162.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2023-12-02T01:13:32.579 INFO:teuthology.orchestra.run.smithi162.stdout:(18/29): librgw2-18.0.0-7631.gbed76814.el9.x86_ 2.6 MB/s | 4.8 MB 00:01 2023-12-02T01:13:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 650 kB/s | 309 kB 00:00 2023-12-02T01:13:33.046 INFO:teuthology.orchestra.run.smithi162.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 417 kB/s | 24 kB 00:00 2023-12-02T01:13:33.105 INFO:teuthology.orchestra.run.smithi162.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2023-12-02T01:13:33.164 INFO:teuthology.orchestra.run.smithi162.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2023-12-02T01:13:33.739 INFO:teuthology.orchestra.run.smithi162.stdout:(23/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.4 MB/s | 837 kB 00:00 2023-12-02T01:13:33.889 INFO:teuthology.orchestra.run.smithi162.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2023-12-02T01:13:35.198 INFO:teuthology.orchestra.run.smithi162.stdout:(25/29): thrift-0.15.0-2.el9.x86_64.rpm 1.2 MB/s | 1.6 MB 00:01 2023-12-02T01:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout:(26/29): libarrow-9.0.0-9.el9.x86_64.rpm 1.4 MB/s | 4.4 MB 00:03 2023-12-02T01:13:36.865 INFO:teuthology.orchestra.run.smithi162.stdout:(27/29): librados2-18.0.0-7631.gbed76814.el9.x8 2.0 MB/s | 3.3 MB 00:01 2023-12-02T01:13:37.948 INFO:teuthology.orchestra.run.smithi162.stdout:(28/29): librbd1-18.0.0-7631.gbed76814.el9.x86_ 1.4 MB/s | 3.0 MB 00:02 2023-12-02T01:13:44.039 INFO:teuthology.orchestra.run.smithi162.stdout:(29/29): ceph-common-18.0.0-7631.gbed76814.el9. 1.1 MB/s | 19 MB 00:17 2023-12-02T01:13:44.045 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:13:44.045 INFO:teuthology.orchestra.run.smithi162.stdout:Total 3.2 MB/s | 55 MB 00:17 2023-12-02T01:13:44.045 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:13:44.081 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:13:44.081 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:13:44.548 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:13:44.548 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:13:45.899 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:13:45.983 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-02T01:13:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-02T01:13:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-02T01:13:46.405 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-02T01:13:46.467 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T01:13:46.559 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T01:13:46.652 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 6/31 2023-12-02T01:13:46.702 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T01:13:46.765 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T01:13:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-02T01:13:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9. 9/31 2023-12-02T01:13:46.970 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 10/31 2023-12-02T01:13:47.005 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 11/31 2023-12-02T01:13:47.053 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 11/31 2023-12-02T01:13:47.154 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-02T01:13:47.189 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-02T01:13:47.235 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-02T01:13:47.287 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-02T01:13:47.316 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-02T01:13:47.349 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-02T01:13:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-02T01:13:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 19/31 2023-12-02T01:13:47.769 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 19/31 2023-12-02T01:13:47.800 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 20/31 2023-12-02T01:13:47.836 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-02T01:13:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-02T01:13:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 23/31 2023-12-02T01:13:48.352 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 23/31 2023-12-02T01:13:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 24/31 2023-12-02T01:13:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x8 25/31 2023-12-02T01:13:50.600 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T01:13:50.638 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T01:13:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T01:13:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 27/31 2023-12-02T01:13:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 27/31 2023-12-02T01:13:51.321 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-02T01:13:51.321 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-02T01:13:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:13:51.360 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T01:14:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T01:14:00.583 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2023-12-02T01:14:00.584 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2023-12-02T01:14:00.584 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2023-12-02T01:14:00.584 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2023-12-02T01:14:00.584 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2023-12-02T01:14:00.584 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2023-12-02T01:14:00.584 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2023-12-02T01:14:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:01.391 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 29/31 2023-12-02T01:14:01.415 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 29/31 2023-12-02T01:14:01.415 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:14:01.415 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-02T01:14:01.415 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T01:14:01.415 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T01:14:01.415 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:01.417 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T01:14:01.510 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T01:14:01.510 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T01:14:02.623 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T01:14:02.623 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 1/31 2023-12-02T01:14:02.623 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2/31 2023-12-02T01:14:02.623 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 3/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 4/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 6/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9. 8/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x8 9/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 10/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 11/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 12/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 13/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-02T01:14:02.624 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-02T01:14:02.625 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-02T01:14:02.626 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T01:14:02.626 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-02T01:14:02.626 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 30/31 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-02T01:14:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:03.723 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:04.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2023-12-02T01:14:04.642 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:40 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:04.789 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:04.789 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:14:04.789 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:04.789 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:18.0.0-7631.gbed76814.el9 ceph 41 M 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:18.0.0-7631.gbed76814.el9 ceph 164 k 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2023-12-02T01:14:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:04.791 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 42 M 2023-12-02T01:14:04.791 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 173 M 2023-12-02T01:14:04.791 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:14:04.969 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-12-02T01:14:05.027 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.7 MB/s | 217 kB 00:00 2023-12-02T01:14:05.077 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-12-02T01:14:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): libcephsqlite-18.0.0-7631.gbed76814.el9. 526 kB/s | 164 kB 00:00 2023-12-02T01:14:05.119 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-02T01:14:20.067 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): ceph-test-18.0.0-7631.gbed76814.el9.x86_ 2.7 MB/s | 41 MB 00:15 2023-12-02T01:14:20.070 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:14:20.071 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.8 MB/s | 42 MB 00:15 2023-12-02T01:14:20.071 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:14:20.091 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:14:20.092 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:14:20.442 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:14:20.443 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:14:21.001 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:14:21.082 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-02T01:14:21.127 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-02T01:14:21.166 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-02T01:14:21.210 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-02T01:14:21.237 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 5/6 2023-12-02T01:14:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 5/6 2023-12-02T01:14:25.199 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 6/6 2023-12-02T01:14:27.357 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 6/6 2023-12-02T01:14:27.357 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 1/6 2023-12-02T01:14:27.357 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 2/6 2023-12-02T01:14:27.357 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-02T01:14:27.357 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-02T01:14:27.357 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-02T01:14:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-02T01:14:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:27.747 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:14:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2023-12-02T01:14:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-02T01:14:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-02T01:14:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-02T01:14:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:27.748 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:28.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2023-12-02T01:14:28.634 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:04 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:28.788 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:28.790 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2023-12-02T01:14:28.790 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:14:28.790 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2023-12-02T01:14:28.790 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:14:28.790 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:18.0.0-7631.gbed76814.el9 ceph 6.6 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:18.0.0-7631.gbed76814.el9 ceph 2.1 M 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:18.0.0-7631.gbed76814.el9 ceph 1.5 M 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 243 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:18.0.0-7631.gbed76814.el9 ceph 4.4 M 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 18 M 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-02T01:14:28.791 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-02T01:14:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout:Install 36 Packages 2023-12-02T01:14:28.793 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:28.794 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 30 M 2023-12-02T01:14:28.794 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 111 M 2023-12-02T01:14:28.794 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:14:29.546 INFO:teuthology.orchestra.run.smithi162.stdout:(1/36): ceph-18.0.0-7631.gbed76814.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2023-12-02T01:14:30.405 INFO:teuthology.orchestra.run.smithi162.stdout:(2/36): ceph-mgr-18.0.0-7631.gbed76814.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2023-12-02T01:14:32.472 INFO:teuthology.orchestra.run.smithi162.stdout:(3/36): ceph-mds-18.0.0-7631.gbed76814.el9.x86_ 648 kB/s | 2.1 MB 00:03 2023-12-02T01:14:32.789 INFO:teuthology.orchestra.run.smithi162.stdout:(4/36): ceph-mon-18.0.0-7631.gbed76814.el9.x86_ 1.4 MB/s | 4.4 MB 00:03 2023-12-02T01:14:32.814 INFO:teuthology.orchestra.run.smithi162.stdout:(5/36): ceph-mgr-modules-core-18.0.0-7631.gbed7 711 kB/s | 243 kB 00:00 2023-12-02T01:14:32.948 INFO:teuthology.orchestra.run.smithi162.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-12-02T01:14:32.973 INFO:teuthology.orchestra.run.smithi162.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2023-12-02T01:14:32.997 INFO:teuthology.orchestra.run.smithi162.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 650 kB/s | 32 kB 00:00 2023-12-02T01:14:33.022 INFO:teuthology.orchestra.run.smithi162.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 970 kB/s | 47 kB 00:00 2023-12-02T01:14:33.039 INFO:teuthology.orchestra.run.smithi162.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 928 kB/s | 39 kB 00:00 2023-12-02T01:14:33.072 INFO:teuthology.orchestra.run.smithi162.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-02T01:14:33.123 INFO:teuthology.orchestra.run.smithi162.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2023-12-02T01:14:33.148 INFO:teuthology.orchestra.run.smithi162.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2023-12-02T01:14:33.174 INFO:teuthology.orchestra.run.smithi162.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-02T01:14:33.199 INFO:teuthology.orchestra.run.smithi162.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-02T01:14:33.224 INFO:teuthology.orchestra.run.smithi162.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-12-02T01:14:33.433 INFO:teuthology.orchestra.run.smithi162.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 726 kB/s | 151 kB 00:00 2023-12-02T01:14:33.474 INFO:teuthology.orchestra.run.smithi162.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2023-12-02T01:14:33.616 INFO:teuthology.orchestra.run.smithi162.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2023-12-02T01:14:33.778 INFO:teuthology.orchestra.run.smithi162.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2023-12-02T01:14:33.817 INFO:teuthology.orchestra.run.smithi162.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 255 kB/s | 11 kB 00:00 2023-12-02T01:14:33.843 INFO:teuthology.orchestra.run.smithi162.stdout:(22/36): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2023-12-02T01:14:33.868 INFO:teuthology.orchestra.run.smithi162.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 359 kB/s | 18 kB 00:00 2023-12-02T01:14:33.935 INFO:teuthology.orchestra.run.smithi162.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 474 kB/s | 19 kB 00:00 2023-12-02T01:14:33.977 INFO:teuthology.orchestra.run.smithi162.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 485 kB/s | 20 kB 00:00 2023-12-02T01:14:34.010 INFO:teuthology.orchestra.run.smithi162.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 138 kB/s | 23 kB 00:00 2023-12-02T01:14:34.052 INFO:teuthology.orchestra.run.smithi162.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2023-12-02T01:14:34.086 INFO:teuthology.orchestra.run.smithi162.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-02T01:14:34.120 INFO:teuthology.orchestra.run.smithi162.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 496 kB/s | 16 kB 00:00 2023-12-02T01:14:34.195 INFO:teuthology.orchestra.run.smithi162.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2023-12-02T01:14:34.221 INFO:teuthology.orchestra.run.smithi162.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 895 kB/s | 90 kB 00:00 2023-12-02T01:14:34.237 INFO:teuthology.orchestra.run.smithi162.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 851 kB/s | 36 kB 00:00 2023-12-02T01:14:34.355 INFO:teuthology.orchestra.run.smithi162.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2023-12-02T01:14:34.388 INFO:teuthology.orchestra.run.smithi162.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 601 kB/s | 20 kB 00:00 2023-12-02T01:14:34.430 INFO:teuthology.orchestra.run.smithi162.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2023-12-02T01:14:36.080 INFO:teuthology.orchestra.run.smithi162.stdout:(36/36): ceph-osd-18.0.0-7631.gbed76814.el9.x86 3.1 MB/s | 18 MB 00:05 2023-12-02T01:14:36.087 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:14:36.087 INFO:teuthology.orchestra.run.smithi162.stdout:Total 4.1 MB/s | 30 MB 00:07 2023-12-02T01:14:36.100 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:14:36.143 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:14:36.143 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:14:36.557 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:14:36.558 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:14:37.897 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:14:37.966 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-02T01:14:37.998 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-02T01:14:38.027 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-02T01:14:38.106 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-02T01:14:38.151 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-02T01:14:38.184 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-02T01:14:38.276 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-02T01:14:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-02T01:14:38.386 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-02T01:14:38.458 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-02T01:14:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-02T01:14:38.559 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-02T01:14:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-02T01:14:38.664 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-02T01:14:38.849 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-02T01:14:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-02T01:14:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-02T01:14:38.995 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-02T01:14:39.025 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-02T01:14:39.103 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-02T01:14:39.136 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T01:14:39.169 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-02T01:14:39.209 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-02T01:14:39.244 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-02T01:14:39.279 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-02T01:14:39.368 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-02T01:14:39.419 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-02T01:14:39.500 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-02T01:14:39.644 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-02T01:14:39.761 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T01:14:39.923 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9. 31/36 2023-12-02T01:14:39.949 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 32/36 2023-12-02T01:14:39.975 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 32/36 2023-12-02T01:14:39.975 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:14:39.975 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-02T01:14:39.975 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T01:14:39.976 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T01:14:39.976 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:41.492 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 33/36 2023-12-02T01:14:41.519 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 33/36 2023-12-02T01:14:41.519 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:14:41.519 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-02T01:14:41.519 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T01:14:41.519 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T01:14:41.519 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 34/36 2023-12-02T01:14:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 34/36 2023-12-02T01:14:41.999 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:14:41.999 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-02T01:14:42.000 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T01:14:42.000 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T01:14:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:42.202 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 35/36 2023-12-02T01:14:42.225 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 35/36 2023-12-02T01:14:42.225 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:14:42.225 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-02T01:14:42.225 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T01:14:42.225 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T01:14:42.225 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:42.291 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:18.0.0-7631.gbed76814.el9.x86_64 36/36 2023-12-02T01:14:44.209 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:18.0.0-7631.gbed76814.el9.x86_64 36/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:18.0.0-7631.gbed76814.el9.x86_64 1/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 2/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 3/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 4/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 5/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9. 6/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-02T01:14:44.210 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-02T01:14:44.211 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-02T01:14:44.212 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-02T01:14:44.212 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T01:14:44.212 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-02T01:14:44.212 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-02T01:14:44.212 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-02T01:14:44.212 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-02T01:14:44.212 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-02T01:14:45.026 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-02T01:14:45.026 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:45.026 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:14:45.026 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:45.026 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-02T01:14:45.027 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-02T01:14:45.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-02T01:14:45.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-02T01:14:45.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-02T01:14:45.029 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:45.029 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:45.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2023-12-02T01:14:45.937 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:21 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:46.033 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:14:46.082 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:46.083 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:14:46.083 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:46.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2023-12-02T01:14:46.717 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:22 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:46.862 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 776 k 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:14:46.863 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:46.864 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:14:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:46.864 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 776 k 2023-12-02T01:14:46.864 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 790 k 2023-12-02T01:14:46.864 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:14:47.718 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-18.0.0-7631.gbed76814.el9.noarch.rpm 909 kB/s | 776 kB 00:00 2023-12-02T01:14:47.718 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:14:47.719 INFO:teuthology.orchestra.run.smithi162.stdout:Total 907 kB/s | 776 kB 00:00 2023-12-02T01:14:47.719 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:14:47.725 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:14:47.725 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:14:47.731 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:14:47.731 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:14:47.807 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:14:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T01:14:48.357 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T01:14:48.761 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T01:14:49.056 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T01:14:49.056 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:49.057 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:14:49.057 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:14:49.057 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:49.057 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:49.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2023-12-02T01:14:49.845 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:25 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:49.989 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7631.gbed76814.el9 ceph 141 k 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:14:49.990 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:49.991 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 141 k 2023-12-02T01:14:49.991 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 400 k 2023-12-02T01:14:49.992 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:14:50.858 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-18.0.0-7631.gbed768 163 kB/s | 141 kB 00:00 2023-12-02T01:14:50.859 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:14:50.859 INFO:teuthology.orchestra.run.smithi162.stdout:Total 163 kB/s | 141 kB 00:00 2023-12-02T01:14:50.859 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:14:50.868 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:14:50.868 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:14:50.923 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:14:50.924 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:14:51.063 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:14:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T01:14:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T01:14:51.167 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:14:51.167 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-02T01:14:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:52.153 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T01:14:52.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:52.154 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:14:52.154 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:14:52.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:52.154 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:52.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2023-12-02T01:14:52.976 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:28 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:53.072 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:14:53.122 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:53.123 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:14:53.124 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:53.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2023-12-02T01:14:53.764 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:29 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:53.910 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 1.8 M 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 28 k 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 15 k 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-02T01:14:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout:Install 7 Packages 2023-12-02T01:14:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:53.913 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.1 M 2023-12-02T01:14:53.913 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 8.7 M 2023-12-02T01:14:53.913 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:14:54.412 INFO:teuthology.orchestra.run.smithi162.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7631.gbed 63 kB/s | 28 kB 00:00 2023-12-02T01:14:54.462 INFO:teuthology.orchestra.run.smithi162.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7631.gbed7 30 kB/s | 15 kB 00:00 2023-12-02T01:14:54.521 INFO:teuthology.orchestra.run.smithi162.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2023-12-02T01:14:54.621 INFO:teuthology.orchestra.run.smithi162.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 258 kB/s | 41 kB 00:00 2023-12-02T01:14:54.646 INFO:teuthology.orchestra.run.smithi162.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 245 kB/s | 31 kB 00:00 2023-12-02T01:14:54.788 INFO:teuthology.orchestra.run.smithi162.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2023-12-02T01:14:56.047 INFO:teuthology.orchestra.run.smithi162.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7631.gbed76814 867 kB/s | 1.8 MB 00:02 2023-12-02T01:14:56.049 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:14:56.049 INFO:teuthology.orchestra.run.smithi162.stdout:Total 990 kB/s | 2.1 MB 00:02 2023-12-02T01:14:56.050 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:14:56.059 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:14:56.059 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:14:56.128 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:14:56.129 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:14:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:14:56.506 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-02T01:14:56.541 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-02T01:14:56.596 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-02T01:14:56.628 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-02T01:14:56.658 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.n 5/7 2023-12-02T01:14:56.897 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9. 6/7 2023-12-02T01:14:56.926 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 7/7 2023-12-02T01:14:57.521 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 7/7 2023-12-02T01:14:57.521 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9. 1/7 2023-12-02T01:14:57.521 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 2/7 2023-12-02T01:14:57.521 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.n 3/7 2023-12-02T01:14:57.521 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-02T01:14:57.521 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-02T01:14:57.522 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-02T01:14:57.857 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-02T01:14:57.857 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:57.857 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:14:57.857 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:14:57.857 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:14:57.857 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:14:57.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-02T01:14:57.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-02T01:14:57.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-02T01:14:57.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-02T01:14:57.858 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:57.858 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:14:58.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-02T01:14:58.692 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:34 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:14:58.838 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 7.4 M 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-02T01:14:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2023-12-02T01:14:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:14:58.842 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 37 M 2023-12-02T01:14:58.842 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 209 M 2023-12-02T01:14:58.842 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:14:59.072 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 168 kB/s | 32 kB 00:00 2023-12-02T01:14:59.123 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 341 kB/s | 17 kB 00:00 2023-12-02T01:14:59.181 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 584 kB/s | 34 kB 00:00 2023-12-02T01:14:59.348 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.4 MB/s | 3.0 MB 00:00 2023-12-02T01:14:59.507 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): ceph-mgr-diskprediction-local-18.0.0-76 12 MB/s | 7.4 MB 00:00 2023-12-02T01:14:59.716 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 8.9 MB/s | 4.7 MB 00:00 2023-12-02T01:14:59.783 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2023-12-02T01:14:59.809 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 729 kB/s | 217 kB 00:00 2023-12-02T01:15:00.233 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2023-12-02T01:15:00.258 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 436 kB/s | 195 kB 00:00 2023-12-02T01:15:00.283 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2023-12-02T01:15:00.476 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2023-12-02T01:15:00.479 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:00.479 INFO:teuthology.orchestra.run.smithi162.stdout:Total 23 MB/s | 37 MB 00:01 2023-12-02T01:15:00.493 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:00.518 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:00.518 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:00.910 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:00.910 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:01.629 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:01.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-02T01:15:01.755 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-02T01:15:01.849 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-02T01:15:01.891 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-02T01:15:02.072 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T01:15:02.107 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T01:15:02.191 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-02T01:15:02.879 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-02T01:15:02.997 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-02T01:15:04.271 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-02T01:15:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-02T01:15:05.350 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 12/12 2023-12-02T01:15:07.895 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 12/12 2023-12-02T01:15:07.895 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 1/12 2023-12-02T01:15:07.895 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-02T01:15:07.895 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-02T01:15:07.895 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-02T01:15:07.895 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T01:15:07.895 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T01:15:07.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-02T01:15:07.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-02T01:15:07.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-02T01:15:07.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-02T01:15:07.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-02T01:15:08.346 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-02T01:15:08.346 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:08.346 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:08.346 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:15:08.346 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-02T01:15:08.346 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-02T01:15:08.346 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:08.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2023-12-02T01:15:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:45 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:09.491 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 50 k 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-02T01:15:09.492 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2023-12-02T01:15:09.493 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:09.494 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.0 M 2023-12-02T01:15:09.494 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2023-12-02T01:15:09.494 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:09.718 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 139 kB/s | 24 kB 00:00 2023-12-02T01:15:09.744 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 90 kB/s | 18 kB 00:00 2023-12-02T01:15:09.844 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2023-12-02T01:15:09.869 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2023-12-02T01:15:09.903 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2023-12-02T01:15:09.928 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 727 kB/s | 43 kB 00:00 2023-12-02T01:15:10.053 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): ceph-mgr-rook-18.0.0-7631.gbed76814.el9 98 kB/s | 50 kB 00:00 2023-12-02T01:15:10.320 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 77 kB/s | 32 kB 00:00 2023-12-02T01:15:10.345 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 36 kB/s | 15 kB 00:00 2023-12-02T01:15:10.612 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 222 kB/s | 59 kB 00:00 2023-12-02T01:15:10.646 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 333 kB/s | 197 kB 00:00 2023-12-02T01:15:10.880 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 335 kB/s | 90 kB 00:00 2023-12-02T01:15:11.022 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-12-02T01:15:11.024 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:11.024 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2023-12-02T01:15:11.024 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:11.038 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:11.038 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:11.140 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:11.142 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:11.651 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:11.828 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-02T01:15:11.881 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-02T01:15:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-02T01:15:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-02T01:15:12.016 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-02T01:15:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-02T01:15:12.218 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-02T01:15:12.284 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-02T01:15:13.015 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-02T01:15:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-02T01:15:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-02T01:15:13.154 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-02T01:15:13.185 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 13/13 2023-12-02T01:15:13.981 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 13/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 1/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-02T01:15:13.982 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-02T01:15:14.364 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-02T01:15:14.365 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-02T01:15:14.365 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-02T01:15:14.365 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-02T01:15:14.365 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:14.365 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:14.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2023-12-02T01:15:15.207 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:51 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:15.356 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:15.356 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2023-12-02T01:15:15.356 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:15:15.356 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2023-12-02T01:15:15.356 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 133 k 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2023-12-02T01:15:15.357 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:15.358 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 6.8 M 2023-12-02T01:15:15.358 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 33 M 2023-12-02T01:15:15.358 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:15.926 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.0 MB/s | 228 kB 00:00 2023-12-02T01:15:16.167 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2023-12-02T01:15:16.234 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): ceph-mgr-cephadm-18.0.0-7631.gbed76814.e 250 kB/s | 133 kB 00:00 2023-12-02T01:15:16.518 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2023-12-02T01:15:16.576 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 843 kB/s | 548 kB 00:00 2023-12-02T01:15:16.618 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 133 kB/s | 51 kB 00:00 2023-12-02T01:15:16.620 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:16.620 INFO:teuthology.orchestra.run.smithi162.stdout:Total 5.4 MB/s | 6.8 MB 00:01 2023-12-02T01:15:16.620 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:16.630 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:16.630 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:16.721 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:16.722 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:17.175 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-02T01:15:17.236 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-02T01:15:17.756 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-02T01:15:17.841 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-02T01:15:17.909 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-02T01:15:17.933 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 6/6 2023-12-02T01:15:18.767 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 6/6 2023-12-02T01:15:18.767 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/6 2023-12-02T01:15:18.767 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-02T01:15:18.767 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-02T01:15:18.767 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-02T01:15:18.768 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-02T01:15:19.079 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-02T01:15:19.080 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:19.080 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:19.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2023-12-02T01:15:19.929 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:55 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:20.076 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:20.076 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:20.076 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:15:20.076 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:20.076 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:20.076 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:18.0.0-7631.gbed76814.el9 ceph 859 k 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:20.077 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 938 k 2023-12-02T01:15:20.078 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.7 M 2023-12-02T01:15:20.078 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:20.281 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 386 kB/s | 79 kB 00:00 2023-12-02T01:15:21.865 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-18.0.0-7631.gbed76814.el9.x86_ 481 kB/s | 859 kB 00:01 2023-12-02T01:15:21.867 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:21.867 INFO:teuthology.orchestra.run.smithi162.stdout:Total 524 kB/s | 938 kB 00:01 2023-12-02T01:15:21.867 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:21.890 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:21.890 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:21.970 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:21.970 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:22.306 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-02T01:15:22.372 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2/2 2023-12-02T01:15:23.117 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2/2 2023-12-02T01:15:23.117 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/2 2023-12-02T01:15:23.420 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-02T01:15:23.421 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:23.421 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:23.421 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-02T01:15:23.421 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:23.421 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:23.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-volume 2023-12-02T01:15:24.274 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:00 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 262 k 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 262 k 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.2 M 2023-12-02T01:15:24.420 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:25.529 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-volume-18.0.0-7631.gbed76814.el9.noarch.rp 237 kB/s | 262 kB 00:01 2023-12-02T01:15:25.529 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:25.530 INFO:teuthology.orchestra.run.smithi162.stdout:Total 236 kB/s | 262 kB 00:01 2023-12-02T01:15:25.530 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:25.539 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:25.539 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:25.585 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:25.585 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:25.864 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T01:15:25.880 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T01:15:25.880 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:15:25.880 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-02T01:15:25.880 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T01:15:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:26.920 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T01:15:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:26.921 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:27.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2023-12-02T01:15:27.846 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:03 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:27.993 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:27.994 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:27.994 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2023-12-02T01:15:27.994 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:27.994 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:27.994 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:18.0.0-7631.gbed76814.el9 ceph 127 k 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 127 k 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2023-12-02T01:15:27.995 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:28.601 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-18.0.0-7631.gbed76814.el9.x86_64 209 kB/s | 127 kB 00:00 2023-12-02T01:15:28.602 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:28.602 INFO:teuthology.orchestra.run.smithi162.stdout:Total 208 kB/s | 127 kB 00:00 2023-12-02T01:15:28.602 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:28.608 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:28.608 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:28.666 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:28.666 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:28.942 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:29.745 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:15:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:29.746 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:30.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2023-12-02T01:15:30.608 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:06 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:30.707 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:15:30.757 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:30.758 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:15:30.758 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:30.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2023-12-02T01:15:31.407 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:07 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:31.555 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:31.555 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2023-12-02T01:15:31.555 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:18.0.0-7631.gbed76814.el9 ceph 31 k 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 31 k 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 139 k 2023-12-02T01:15:31.556 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:32.073 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-18.0.0-7631.gbed76814.el9.x86_6 60 kB/s | 31 kB 00:00 2023-12-02T01:15:32.073 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:32.074 INFO:teuthology.orchestra.run.smithi162.stdout:Total 60 kB/s | 31 kB 00:00 2023-12-02T01:15:32.074 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:32.078 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:32.079 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:32.109 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:32.109 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:32.306 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:32.975 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:32.975 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:32.975 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:32.975 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:15:32.975 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:32.975 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:33.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2023-12-02T01:15:33.747 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:09 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:33.843 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:15:33.892 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:33.893 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:15:33.893 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:33.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2023-12-02T01:15:34.547 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:10 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:34.644 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:15:34.694 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:34.695 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:15:34.695 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:34.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2023-12-02T01:15:35.353 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:11 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:35.449 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:15:35.500 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:35.501 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:15:35.501 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:35.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2023-12-02T01:15:36.140 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:12 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:36.235 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:15:36.287 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:36.288 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:15:36.288 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:36.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2023-12-02T01:15:36.944 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:12 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:37.041 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:15:37.092 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:37.093 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:15:37.093 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:37.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2023-12-02T01:15:37.740 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:13 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:37.834 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T01:15:37.884 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:37.884 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2023-12-02T01:15:37.885 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:37.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2023-12-02T01:15:38.557 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:14 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:38.703 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:38.703 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:38.703 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:18.0.0-7631.gbed76814.el9 ceph 85 k 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2023-12-02T01:15:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:39.284 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-18.0.0-7631.gbed76814.el9.x86_64.rpm 147 kB/s | 85 kB 00:00 2023-12-02T01:15:39.285 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:39.285 INFO:teuthology.orchestra.run.smithi162.stdout:Total 147 kB/s | 85 kB 00:00 2023-12-02T01:15:39.285 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:39.292 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:39.292 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:39.349 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:39.349 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:39.487 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:39.600 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:40.021 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:40.365 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:40.365 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:40.365 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:40.365 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:15:40.365 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:40.365 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:40.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2023-12-02T01:15:41.220 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:17 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:41.367 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:41.367 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.0 M 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.0 M 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 12 M 2023-12-02T01:15:41.368 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:44.330 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-18.0.0-7631.gbed76814.el9.x86_64.rpm 1.0 MB/s | 3.0 MB 00:02 2023-12-02T01:15:44.330 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:44.331 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.0 MB/s | 3.0 MB 00:02 2023-12-02T01:15:44.331 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:44.342 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:44.342 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:44.417 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:44.417 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:44.828 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:44.889 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:44.914 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:44.914 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T01:15:44.914 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-02T01:15:44.914 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T01:15:44.914 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T01:15:44.915 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:46.082 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:15:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:46.082 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:46.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2023-12-02T01:15:46.914 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:22 ago on Sat 02 Dec 2023 01:13:24 AM UTC. 2023-12-02T01:15:47.060 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 170 k 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 170 k 2023-12-02T01:15:47.061 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 494 k 2023-12-02T01:15:47.062 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-02T01:15:47.318 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-18.0.0-7631.gbed76814.el9.x86_64.rpm 664 kB/s | 170 kB 00:00 2023-12-02T01:15:47.318 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-02T01:15:47.318 INFO:teuthology.orchestra.run.smithi162.stdout:Total 661 kB/s | 170 kB 00:00 2023-12-02T01:15:47.319 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-02T01:15:47.327 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-02T01:15:47.327 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-02T01:15:47.381 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-02T01:15:47.381 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-02T01:15:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-02T01:15:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:48.078 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:48.382 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T01:15:48.382 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:48.382 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-02T01:15:48.382 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T01:15:48.382 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-02T01:15:48.382 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-02T01:15:48.645 DEBUG:teuthology.parallel:result is None 2023-12-02T01:15:48.645 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T01:15:48.804 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-02T01:15:48.825 INFO:teuthology.orchestra.run.smithi162.stdout:18.0.0-7631.gbed76814.el9 2023-12-02T01:15:48.825 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7631.gbed76814.el9 2023-12-02T01:15:48.825 INFO:teuthology.task.install:The correct ceph version 18.0.0-7631.gbed76814 is installed. 2023-12-02T01:15:48.826 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-02T01:15:48.826 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-02T01:15:48.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-02T01:15:48.893 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-02T01:15:48.894 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-02T01:15:48.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2023-12-02T01:15:48.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-02T01:15:49.027 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-02T01:15:49.027 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-02T01:15:49.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-02T01:15:49.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-02T01:15:49.161 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-02T01:15:49.161 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-02T01:15:49.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2023-12-02T01:15:49.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-02T01:15:49.295 INFO:teuthology.run_tasks:Running task workunit... 2023-12-02T01:15:49.309 INFO:tasks.workunit:Pulling workunits from ref bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T01:15:49.309 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-02T01:15:49.309 INFO:tasks.workunit:timeout=3h 2023-12-02T01:15:49.309 INFO:tasks.workunit:cleanup=True 2023-12-02T01:15:49.310 DEBUG:teuthology.orchestra.run.smithi162:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-02T01:15:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T01:15:49.350 INFO:teuthology.orchestra.run.smithi162.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-02T01:15:49.350 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-02T01:15:49.405 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-02T01:15:49.405 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-02T01:15:49.460 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T01:15:49.514 INFO:tasks.workunit.client.0.smithi162.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-02T01:17:02.684 INFO:tasks.workunit.client.0.smithi162.stderr:Note: switching to 'bed7681480ccb0192b91a35e691658867e1e96b7'. 2023-12-02T01:17:02.684 INFO:tasks.workunit.client.0.smithi162.stderr: 2023-12-02T01:17:02.684 INFO:tasks.workunit.client.0.smithi162.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:state without impacting any branches by switching back to a branch. 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: git switch -c 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:Or undo this operation with: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: git switch - 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr: 2023-12-02T01:17:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:HEAD is now at bed7681480c Merge pull request #54648 from rhcs-dashboard/fs-rename-fix 2023-12-02T01:17:02.688 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-02T01:17:02.744 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-02T01:17:02.744 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-02T01:17:02.799 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2023-12-02T01:17:02.800 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2023-12-02T01:17:02.800 DEBUG:teuthology.orchestra.run.smithi162:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2023-12-02T01:17:02.868 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-12-02T01:17:02.873 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-12-02T01:17:02.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-02T01:17:02.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-02T01:17:02.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2023-12-02T01:17:02.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2023-12-02T01:17:02.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2023-12-02T01:17:02.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2023-12-02T01:17:02.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2023-12-02T01:17:02.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none ' 2023-12-02T01:17:02.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2023-12-02T01:17:02.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-12-02T01:17:02.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2023-12-02T01:17:02.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:17:02.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:17:02.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:17:02.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:17:02.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:17:02.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:17:02.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:17:02.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:17:02.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:17:02.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:17:02.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:17:02.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:17:02.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:17:02.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:17:02.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:17:02.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:17:02.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:17:02.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-12-02T01:17:02.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-12-02T01:17:02.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:17:02.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:17:02.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:17:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80181 2023-12-02T01:17:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:17:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:17:02.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2023-12-02T01:17:02.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-12-02T01:17:02.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:17:02.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:17:02.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80181 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2023-12-02T01:17:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2023-12-02T01:17:02.893 INFO:tasks.workunit.client.0.smithi162.stdout:Dir: td/osd-mapper 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2023-12-02T01:17:02.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2023-12-02T01:17:02.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2023-12-02T01:17:02.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2023-12-02T01:17:04.572 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 819f62a4-8f8b-48cd-90ce-ba9429c124f8 2023-12-02T01:17:04.806 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:17:04.890 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:04.888+0000 7fe23bbe2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:04.892 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:04.890+0000 7fe23bbe2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:04.893 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:04.892+0000 7fe23bbe2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:04.893 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:04.892+0000 7fe23bbe2500 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-12-02T01:17:04.893 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:04.892+0000 7fe23bbe2500 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-12-02T01:17:04.893 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:04.892+0000 7fe23bbe2500 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-12-02T01:17:05.000 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:04.998+0000 7fe23bbe2500 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2023-12-02T01:17:10.781 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2023-12-02T01:17:11.121 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2023-12-02T01:17:11.149 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:11.147+0000 7fd16878b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:11.151 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:11.149+0000 7fd16878b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:11.152 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:11.150+0000 7fd16878b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:11.389 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:17:11.679 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:11.678+0000 7fd16878b500 -1 Falling back to public interface 2023-12-02T01:17:12.655 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:17:13.924 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:17:14.014 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:14.013+0000 7fd16878b500 -1 osd.0 0 log_to_monitors true 2023-12-02T01:17:15.198 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:17:16.478 INFO:tasks.workunit.client.0.smithi162.stdout:4 2023-12-02T01:17:17.772 INFO:tasks.workunit.client.0.smithi162.stdout:5 2023-12-02T01:17:19.051 INFO:tasks.workunit.client.0.smithi162.stdout:6 2023-12-02T01:17:20.327 INFO:tasks.workunit.client.0.smithi162.stdout:7 2023-12-02T01:17:21.607 INFO:tasks.workunit.client.0.smithi162.stdout:8 2023-12-02T01:17:22.924 INFO:tasks.workunit.client.0.smithi162.stdout:9 2023-12-02T01:17:24.197 INFO:tasks.workunit.client.0.smithi162.stdout:10 2023-12-02T01:17:25.479 INFO:tasks.workunit.client.0.smithi162.stdout:11 2023-12-02T01:17:26.796 INFO:tasks.workunit.client.0.smithi162.stdout:12 2023-12-02T01:17:28.144 INFO:tasks.workunit.client.0.smithi162.stdout:13 2023-12-02T01:17:29.632 INFO:tasks.workunit.client.0.smithi162.stdout:14 2023-12-02T01:17:31.349 INFO:tasks.workunit.client.0.smithi162.stdout:15 2023-12-02T01:17:32.700 INFO:tasks.workunit.client.0.smithi162.stdout:16 2023-12-02T01:17:34.503 INFO:tasks.workunit.client.0.smithi162.stdout:17 2023-12-02T01:17:35.780 INFO:tasks.workunit.client.0.smithi162.stdout:18 2023-12-02T01:17:37.354 INFO:tasks.workunit.client.0.smithi162.stdout:19 2023-12-02T01:17:38.669 INFO:tasks.workunit.client.0.smithi162.stdout:20 2023-12-02T01:17:38.941 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3991674115,v1:127.0.0.1:6803/3991674115] [v2:127.0.0.1:6804/3991674115,v1:127.0.0.1:6805/3991674115] exists,up 819f62a4-8f8b-48cd-90ce-ba9429c124f8 2023-12-02T01:17:38.944 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 7de2716a-8c41-4021-8eaa-98fc79ebcb78 2023-12-02T01:17:39.409 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:17:39.450 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:39.449+0000 7fb75a60d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:39.494 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:39.492+0000 7fb75a60d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:39.495 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:39.493+0000 7fb75a60d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:39.495 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:39.494+0000 7fb75a60d500 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-12-02T01:17:39.495 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:39.494+0000 7fb75a60d500 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-12-02T01:17:39.495 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:39.494+0000 7fb75a60d500 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-12-02T01:17:39.571 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:39.570+0000 7fb75a60d500 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2023-12-02T01:17:45.619 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2023-12-02T01:17:45.966 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2023-12-02T01:17:45.993 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:45.992+0000 7fd5a439b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:46.016 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:46.015+0000 7fd5a439b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:46.017 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:46.016+0000 7fd5a439b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:17:46.275 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:17:46.557 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:46.556+0000 7fd5a439b500 -1 Falling back to public interface 2023-12-02T01:17:47.542 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:17:48.814 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:17:48.958 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:17:48.957+0000 7fd5a439b500 -1 osd.1 0 log_to_monitors true 2023-12-02T01:17:50.082 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:17:51.356 INFO:tasks.workunit.client.0.smithi162.stdout:4 2023-12-02T01:17:52.634 INFO:tasks.workunit.client.0.smithi162.stdout:5 2023-12-02T01:17:53.923 INFO:tasks.workunit.client.0.smithi162.stdout:6 2023-12-02T01:17:55.200 INFO:tasks.workunit.client.0.smithi162.stdout:7 2023-12-02T01:17:56.475 INFO:tasks.workunit.client.0.smithi162.stdout:8 2023-12-02T01:17:57.782 INFO:tasks.workunit.client.0.smithi162.stdout:9 2023-12-02T01:17:59.115 INFO:tasks.workunit.client.0.smithi162.stdout:10 2023-12-02T01:18:00.404 INFO:tasks.workunit.client.0.smithi162.stdout:11 2023-12-02T01:18:01.705 INFO:tasks.workunit.client.0.smithi162.stdout:12 2023-12-02T01:18:03.482 INFO:tasks.workunit.client.0.smithi162.stdout:13 2023-12-02T01:18:05.168 INFO:tasks.workunit.client.0.smithi162.stdout:14 2023-12-02T01:18:06.795 INFO:tasks.workunit.client.0.smithi162.stdout:15 2023-12-02T01:18:08.189 INFO:tasks.workunit.client.0.smithi162.stdout:16 2023-12-02T01:18:09.714 INFO:tasks.workunit.client.0.smithi162.stdout:17 2023-12-02T01:18:11.429 INFO:tasks.workunit.client.0.smithi162.stdout:18 2023-12-02T01:18:12.864 INFO:tasks.workunit.client.0.smithi162.stdout:19 2023-12-02T01:18:13.140 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/935373202,v1:127.0.0.1:6811/935373202] [v2:127.0.0.1:6812/935373202,v1:127.0.0.1:6813/935373202] exists,up 7de2716a-8c41-4021-8eaa-98fc79ebcb78 2023-12-02T01:18:13.144 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 bed88b9b-517a-4560-b651-5206fe992d0b 2023-12-02T01:18:13.506 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:18:13.548 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:13.546+0000 7f16db71c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:18:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:13.589+0000 7f16db71c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:18:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:13.590+0000 7f16db71c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:18:13.592 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:13.590+0000 7f16db71c500 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-12-02T01:18:13.592 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:13.590+0000 7f16db71c500 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-12-02T01:18:13.592 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:13.590+0000 7f16db71c500 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-12-02T01:18:13.668 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:13.667+0000 7f16db71c500 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2023-12-02T01:18:19.765 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2023-12-02T01:18:20.151 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2023-12-02T01:18:20.178 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:20.176+0000 7f5a7411e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:18:20.180 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:20.178+0000 7f5a7411e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:18:20.181 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:20.179+0000 7f5a7411e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:18:20.417 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:18:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:20.700+0000 7f5a7411e500 -1 Falling back to public interface 2023-12-02T01:18:21.720 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:18:22.994 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:18:23.109 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:18:23.105+0000 7f5a7411e500 -1 osd.2 0 log_to_monitors true 2023-12-02T01:18:24.315 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:18:25.595 INFO:tasks.workunit.client.0.smithi162.stdout:4 2023-12-02T01:18:27.010 INFO:tasks.workunit.client.0.smithi162.stdout:5 2023-12-02T01:18:28.286 INFO:tasks.workunit.client.0.smithi162.stdout:6 2023-12-02T01:18:29.562 INFO:tasks.workunit.client.0.smithi162.stdout:7 2023-12-02T01:18:30.992 INFO:tasks.workunit.client.0.smithi162.stdout:8 2023-12-02T01:18:32.266 INFO:tasks.workunit.client.0.smithi162.stdout:9 2023-12-02T01:18:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:10 2023-12-02T01:18:34.992 INFO:tasks.workunit.client.0.smithi162.stdout:11 2023-12-02T01:18:36.580 INFO:tasks.workunit.client.0.smithi162.stdout:12 2023-12-02T01:18:38.036 INFO:tasks.workunit.client.0.smithi162.stdout:13 2023-12-02T01:18:39.313 INFO:tasks.workunit.client.0.smithi162.stdout:14 2023-12-02T01:18:41.044 INFO:tasks.workunit.client.0.smithi162.stdout:15 2023-12-02T01:18:42.730 INFO:tasks.workunit.client.0.smithi162.stdout:16 2023-12-02T01:18:44.423 INFO:tasks.workunit.client.0.smithi162.stdout:17 2023-12-02T01:18:46.161 INFO:tasks.workunit.client.0.smithi162.stdout:18 2023-12-02T01:18:46.433 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/483475126,v1:127.0.0.1:6819/483475126] [v2:127.0.0.1:6820/483475126,v1:127.0.0.1:6821/483475126] exists,up bed88b9b-517a-4560-b651-5206fe992d0b 2023-12-02T01:18:46.904 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test' created 2023-12-02T01:18:48.947 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836504 2023-12-02T01:18:50.493 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672972 2023-12-02T01:18:50.785 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509445 2023-12-02T01:18:53.268 INFO:tasks.workunit.client.0.smithi162.stdout:standard_scrub_cluster: dbg: test pool is test 1 2023-12-02T01:18:53.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2023-12-02T01:18:53.363 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: { 2023-12-02T01:18:53.364 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2023-12-02T01:18:53.364 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:53.378 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: { 2023-12-02T01:18:53.378 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2023-12-02T01:18:53.378 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:53.391 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2: { 2023-12-02T01:18:53.391 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2023-12-02T01:18:53.391 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:53.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2023-12-02T01:18:53.531 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: { 2023-12-02T01:18:53.531 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2023-12-02T01:18:53.531 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:53.545 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: { 2023-12-02T01:18:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2023-12-02T01:18:53.546 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:53.559 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2: { 2023-12-02T01:18:53.559 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2023-12-02T01:18:53.559 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:53.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2023-12-02T01:18:53.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2023-12-02T01:18:53.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2023-12-02T01:18:53.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2023-12-02T01:18:53.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2023-12-02T01:18:53.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-12-02T01:18:53.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-12-02T01:18:53.951 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap01 2023-12-02T01:18:54.286 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap02 2023-12-02T01:18:54.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2023-12-02T01:18:54.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-12-02T01:18:54.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-12-02T01:18:54.789 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap13 2023-12-02T01:18:54.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2023-12-02T01:18:54.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-12-02T01:18:54.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-12-02T01:18:55.282 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap24 2023-12-02T01:18:55.501 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap25 2023-12-02T01:18:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 25377 2023-12-02T01:18:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2023-12-02T01:18:55.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-12-02T01:18:55.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2023-12-02T01:18:56.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2023-12-02T01:18:56.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-12-02T01:18:56.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2023-12-02T01:18:56.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2023-12-02T01:18:56.339 INFO:tasks.workunit.client.0.smithi162.stdout:pgid is 1.3 (primary: osd.1) 2023-12-02T01:18:56.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2023-12-02T01:18:56.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2023-12-02T01:18:56.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:294: set_query_debug: local pgid=1.3 2023-12-02T01:18:56.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: ceph pg dump pgs_brief 2023-12-02T01:18:56.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2023-12-02T01:18:56.591 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs_brief 2023-12-02T01:18:56.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: local prim_osd=1 2023-12-02T01:18:56.603 INFO:tasks.workunit.client.0.smithi162.stdout:Setting scrub debug data. Primary for 1.3 is 1 2023-12-02T01:18:56.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2023-12-02T01:18:56.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: get_asok_path osd.1 2023-12-02T01:18:56.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:18:56.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:18:56.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:18:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:18:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:18:56.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80181/ceph-osd.1.asok 2023-12-02T01:18:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: CEPH_ARGS= 2023-12-02T01:18:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.80181/ceph-osd.1.asok scrubdebug 1.3 set sessions 2023-12-02T01:18:56.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:18:56.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout:{"success":true}{ 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "objxxx", 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: "clones": [ 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 1, 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap01" 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:18:56.721 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap02" 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2023-12-02T01:18:56.722 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap13" 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 4, 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap24" 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2023-12-02T01:18:56.723 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap25" 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: "id": "head", 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [], 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2023-12-02T01:18:56.724 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:56.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2023-12-02T01:18:56.815 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:18:56.815 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2023-12-02T01:18:56.815 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:18:56.815 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:18:56.815 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:18:56.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2023-12-02T01:18:59.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2023-12-02T01:19:00.083 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:19:00.083 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2023-12-02T01:18:56.951505+0000 26'7 26:37 [1,2,0] 1 [1,2,0] 1 26'7 2023-12-02T01:18:56.951407+0000 26'7 2023-12-02T01:18:56.951407+0000 0 1 periodic scrub scheduled @ 2023-12-03T01:18:56.951407+0000 4 0 2023-12-02T01:19:00.084 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:18:48.388356+0000 0'0 26:25 [0,1,2] 0 [0,1,2] 0 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:00.084 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:18:47.690019+0000 0'0 26:25 [2,0,1] 2 [2,0,1] 2 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:00.084 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:18:47.690172+0000 0'0 26:23 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:00.084 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:00.084 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:19:00.084 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:19:00.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2023-12-02T01:19:00.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2023-12-02T01:19:00.342 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:19:00.342 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2023-12-02T01:18:56.951505+0000 26'7 26:37 [1,2,0] 1 [1,2,0] 1 26'7 2023-12-02T01:18:56.951407+0000 26'7 2023-12-02T01:18:56.951407+0000 0 1 periodic scrub scheduled @ 2023-12-03T01:18:56.951407+0000 4 0 2023-12-02T01:19:00.342 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:18:48.388356+0000 0'0 26:25 [0,1,2] 0 [0,1,2] 0 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:00.342 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:18:47.690019+0000 0'0 26:25 [2,0,1] 2 [2,0,1] 2 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:00.342 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:18:47.690172+0000 0'0 26:23 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:00.342 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:00.342 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:19:00.343 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:19:00.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2023-12-02T01:19:00.767 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2023-12-02T01:19:00.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2023-12-02T01:19:01.144 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2023-12-02T01:19:01.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2023-12-02T01:19:06.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2023-12-02T01:19:06.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-12-02T01:19:06.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:19:06.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:19:06.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:19:06.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:19:06.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:19:06.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:19:06.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:06.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-12-02T01:19:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.490+0000 7f994fa4f800 1 bdev(0x55b569980000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-12-02T01:19:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.490+0000 7f994fa4f800 0 bdev(0x55b569980000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2023-12-02T01:19:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.490+0000 7f994fa4f800 1 bdev(0x55b569980000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.490+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2023-12-02T01:19:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.490+0000 7f994fa4f800 1 bdev(0x55b569980700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-12-02T01:19:06.493 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.490+0000 7f994fa4f800 0 bdev(0x55b569980700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2023-12-02T01:19:06.493 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.491+0000 7f994fa4f800 1 bdev(0x55b569980700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:06.493 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.491+0000 7f994fa4f800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-12-02T01:19:06.493 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.491+0000 7f994fa4f800 1 bluefs mount 2023-12-02T01:19:06.508 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.507+0000 7f994fa4f800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.533+0000 7f994fa4f800 1 bluefs mount shared_bdev_used = 0 2023-12-02T01:19:06.535 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.533+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:06.537 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.535+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.535+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.535+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.535+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.536+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:06.569 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.567+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:06.569 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.567+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2023-12-02T01:19:06.569 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.567+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-12-02T01:19:06.570 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.568+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-12-02T01:19:06.570 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.568+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-12-02T01:19:06.570 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.568+0000 7f994fa4f800 1 freelist init 2023-12-02T01:19:06.570 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.568+0000 7f994fa4f800 1 freelist _read_cfg 2023-12-02T01:19:06.570 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.568+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 22 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.01088e-07 2023-12-02T01:19:06.570 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.569+0000 7f994fa4f800 1 bluefs umount 2023-12-02T01:19:06.570 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.569+0000 7f994fa4f800 1 bdev(0x55b569980700 td/osd-mapper/0/block) close 2023-12-02T01:19:06.754 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.753+0000 7f994fa4f800 1 bdev(0x55b569980700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-12-02T01:19:06.757 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.753+0000 7f994fa4f800 0 bdev(0x55b569980700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2023-12-02T01:19:06.757 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.753+0000 7f994fa4f800 1 bdev(0x55b569980700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:06.757 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.753+0000 7f994fa4f800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-12-02T01:19:06.757 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.753+0000 7f994fa4f800 1 bluefs mount 2023-12-02T01:19:06.757 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.753+0000 7f994fa4f800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:06.761 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.760+0000 7f994fa4f800 1 bluefs mount shared_bdev_used = 27459584 2023-12-02T01:19:06.761 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.760+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:06.764 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:06.765 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:06.762+0000 7f994fa4f800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:07.511 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:07.510+0000 7f994fa4f800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 fe a9 55 c5 d6 7c 4a dd 91 8e |..X.....U..|J...| 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 30 3f ae 2a 3d e6 00 00 00 00 1c 00 00 00 00 00 |0?.*=...........| 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:* 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-12-02T01:19:07.513 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-12-02T01:19:07.514 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 81 9f 62 a4 8f 8b |re..........b...| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 48 cd 90 ce ba 94 29 c1 24 f8 00 00 00 00 00 00 |H.....).$.......| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1c 00 00 00 b1 85 6a 65 c0 26 81 1a 01 00 |........je.&....| 2023-12-02T01:19:07.515 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-12-02T01:19:07.516 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:07.517 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:07.518 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:07.519 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:07.511+0000 7f994fa4f800 1 bluefs umount 2023-12-02T01:19:07.520 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:07.511+0000 7f994fa4f800 1 bdev(0x55b569980700 td/osd-mapper/0/block) close 2023-12-02T01:19:07.757 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:07.756+0000 7f994fa4f800 1 freelist shutdown 2023-12-02T01:19:07.757 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:07.756+0000 7f994fa4f800 1 bdev(0x55b569980000 td/osd-mapper/0/block) close 2023-12-02T01:19:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-12-02T01:19:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 1 bdev(0x559f03c80000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 0 bdev(0x559f03c80000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 1 bdev(0x559f03c80000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 1 bdev(0x559f03c80700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 0 bdev(0x559f03c80700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 1 bdev(0x559f03c80700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-12-02T01:19:08.036 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.034+0000 7fc592857800 1 bluefs mount 2023-12-02T01:19:08.055 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.054+0000 7fc592857800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:08.064 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.063+0000 7fc592857800 1 bluefs mount shared_bdev_used = 0 2023-12-02T01:19:08.064 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.063+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:08.067 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:08.067 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:08.067 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.065+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:08.098 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.097+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:08.099 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.097+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-12-02T01:19:08.099 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.097+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-12-02T01:19:08.099 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.097+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-12-02T01:19:08.099 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.097+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-12-02T01:19:08.099 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.097+0000 7fc592857800 1 freelist init 2023-12-02T01:19:08.099 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.097+0000 7fc592857800 1 freelist _read_cfg 2023-12-02T01:19:08.100 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.098+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.34059e-07 2023-12-02T01:19:08.100 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.098+0000 7fc592857800 1 bluefs umount 2023-12-02T01:19:08.100 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.098+0000 7fc592857800 1 bdev(0x559f03c80700 td/osd-mapper/2/block) close 2023-12-02T01:19:08.297 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.296+0000 7fc592857800 1 bdev(0x559f03c80700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-12-02T01:19:08.298 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.296+0000 7fc592857800 0 bdev(0x559f03c80700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2023-12-02T01:19:08.298 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.296+0000 7fc592857800 1 bdev(0x559f03c80700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:08.298 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.296+0000 7fc592857800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-12-02T01:19:08.298 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.296+0000 7fc592857800 1 bluefs mount 2023-12-02T01:19:08.305 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.304+0000 7fc592857800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:08.313 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.312+0000 7fc592857800 1 bluefs mount shared_bdev_used = 27459584 2023-12-02T01:19:08.313 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.312+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.313+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:08.317 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:08.317 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:08.314+0000 7fc592857800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:09.148 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:09.147+0000 7fc592857800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:09.149 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:09.148+0000 7fc592857800 1 bluefs umount 2023-12-02T01:19:09.149 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:09.148+0000 7fc592857800 1 bdev(0x559f03c80700 td/osd-mapper/2/block) close 2023-12-02T01:19:09.385 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:09.384+0000 7fc592857800 1 freelist shutdown 2023-12-02T01:19:09.385 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:09.384+0000 7fc592857800 1 bdev(0x559f03c80000 td/osd-mapper/2/block) close 2023-12-02T01:19:09.554 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:09.555 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:09.555 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:09.555 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:09.555 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:09.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:09.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2023-12-02T01:19:09.556 INFO:tasks.workunit.client.0.smithi162.stderr:grep: /tmp/oo2.dump: No such file or directory 2023-12-02T01:19:09.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:09.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 fe a9 55 c5 d6 7c 4a dd 91 8e |..X.....U..|J...| 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 30 3f ae 2a 3d e6 02 00 00 00 1c 00 00 00 00 00 |0?.*=...........| 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:* 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-12-02T01:19:10.151 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-12-02T01:19:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 be d8 8b 9b 51 7a |re............Qz| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 45 60 b6 51 52 06 fe 99 2d 0b 00 00 00 00 00 00 |E`.QR...-.......| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1c 00 00 00 f4 85 6a 65 23 74 78 3a 01 00 |........je#tx:..| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2023-12-02T01:19:10.153 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:10.154 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.157 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-12-02T01:19:10.158 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:10.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-12-02T01:19:10.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-12-02T01:19:10.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-12-02T01:19:10.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2023-12-02T01:19:12.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2023-12-02T01:19:12.140 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2023-12-02T01:19:12.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2023-12-02T01:19:12.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:12.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2023-12-02T01:19:12.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2023-12-02T01:19:12.143 INFO:tasks.workunit.client.0.smithi162.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2023-12-02T01:19:12.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-12-02T01:19:12.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 1 bdev(0x56486b838000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 0 bdev(0x56486b838000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 1 bdev(0x56486b838000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 1 bdev(0x56486b838700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 0 bdev(0x56486b838700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 1 bdev(0x56486b838700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-12-02T01:19:12.168 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.166+0000 7ff549039800 1 bluefs mount 2023-12-02T01:19:12.173 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.172+0000 7ff549039800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:12.184 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.182+0000 7ff549039800 1 bluefs mount shared_bdev_used = 0 2023-12-02T01:19:12.184 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.182+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.184+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.184+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.184+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.185+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.189+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.190+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2023-12-02T01:19:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.190+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-12-02T01:19:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.190+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-12-02T01:19:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.190+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-12-02T01:19:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.190+0000 7ff549039800 1 freelist init 2023-12-02T01:19:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.190+0000 7ff549039800 1 freelist _read_cfg 2023-12-02T01:19:12.192 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.190+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 22 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.01088e-07 2023-12-02T01:19:12.192 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.191+0000 7ff549039800 1 bluefs umount 2023-12-02T01:19:12.192 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.191+0000 7ff549039800 1 bdev(0x56486b838700 td/osd-mapper/0/block) close 2023-12-02T01:19:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.428+0000 7ff549039800 1 bdev(0x56486b838700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-12-02T01:19:12.430 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.428+0000 7ff549039800 0 bdev(0x56486b838700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2023-12-02T01:19:12.430 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.428+0000 7ff549039800 1 bdev(0x56486b838700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:12.430 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.428+0000 7ff549039800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-12-02T01:19:12.430 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.428+0000 7ff549039800 1 bluefs mount 2023-12-02T01:19:12.440 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.439+0000 7ff549039800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:12.449 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.448+0000 7ff549039800 1 bluefs mount shared_bdev_used = 9568256 2023-12-02T01:19:12.449 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.448+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:12.453 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:12.453 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:12.453 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.450+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:12.453 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.451+0000 7ff549039800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:12.781 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.779+0000 7ff549039800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:12.784 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-12-02T01:19:12.784 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 fe a9 55 c5 d6 7c 4a dd 91 8e |..X.....U..|J...| 2023-12-02T01:19:12.784 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 30 3f ae 2a 3d e6 00 00 00 00 1c 00 00 00 00 00 |0?.*=...........| 2023-12-02T01:19:12.784 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-12-02T01:19:12.784 INFO:tasks.workunit.client.0.smithi162.stdout:* 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-12-02T01:19:12.785 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-12-02T01:19:12.786 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 81 9f 62 a4 8f 8b |re..........b...| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 48 cd 90 ce ba 94 29 c1 24 f8 00 00 00 00 00 00 |H.....).$.......| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1c 00 00 00 b1 85 6a 65 c0 26 81 1a 01 00 |........je.&....| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:12.787 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:12.788 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:12.789 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:12.790 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:12.791 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.792 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-12-02T01:19:12.792 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.792 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-12-02T01:19:12.792 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.780+0000 7ff549039800 1 bluefs umount 2023-12-02T01:19:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:12.780+0000 7ff549039800 1 bdev(0x56486b838700 td/osd-mapper/0/block) close 2023-12-02T01:19:13.001 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:13.000+0000 7ff549039800 1 freelist shutdown 2023-12-02T01:19:13.001 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:13.000+0000 7ff549039800 1 bdev(0x56486b838000 td/osd-mapper/0/block) close 2023-12-02T01:19:13.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-12-02T01:19:13.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-12-02T01:19:13.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-12-02T01:19:14.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:14.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:14.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-12-02T01:19:14.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-12-02T01:19:14.224 INFO:tasks.workunit.client.0.smithi162.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:14.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-12-02T01:19:14.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.0njoxz5Ua3_the_val 2023-12-02T01:19:14.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:14.226 INFO:tasks.workunit.client.0.smithi162.stdout:Value dumped in: /tmp/tmp.0njoxz5Ua3_the_val 2023-12-02T01:19:14.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.0njoxz5Ua3_the_val' 2023-12-02T01:19:14.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.0njoxz5Ua3_the_val 2023-12-02T01:19:14.883 INFO:tasks.workunit.client.0.smithi162.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-12-02T01:19:15.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:15.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.0njoxz5Ua3_the_val 2023-12-02T01:19:15.279 INFO:tasks.workunit.client.0.smithi162.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-12-02T01:19:15.279 INFO:tasks.workunit.client.0.smithi162.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-12-02T01:19:15.279 INFO:tasks.workunit.client.0.smithi162.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-12-02T01:19:15.279 INFO:tasks.workunit.client.0.smithi162.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-12-02T01:19:15.280 INFO:tasks.workunit.client.0.smithi162.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-12-02T01:19:15.280 INFO:tasks.workunit.client.0.smithi162.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-12-02T01:19:15.280 INFO:tasks.workunit.client.0.smithi162.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-12-02T01:19:15.280 INFO:tasks.workunit.client.0.smithi162.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-12-02T01:19:15.280 INFO:tasks.workunit.client.0.smithi162.stdout:0000073 2023-12-02T01:19:15.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-12-02T01:19:15.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:16.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.0njoxz5Ua3_the_val 2023-12-02T01:19:17.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:17.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:17.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-12-02T01:19:17.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2023-12-02T01:19:17.626 INFO:tasks.workunit.client.0.smithi162.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2023-12-02T01:19:17.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-12-02T01:19:17.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-12-02T01:19:17.643 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.641+0000 7fa617010800 1 bdev(0x561d2e780000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-12-02T01:19:17.643 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.641+0000 7fa617010800 0 bdev(0x561d2e780000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2023-12-02T01:19:17.643 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.641+0000 7fa617010800 1 bdev(0x561d2e780000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:17.643 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.641+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2023-12-02T01:19:17.644 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.641+0000 7fa617010800 1 bdev(0x561d2e780700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-12-02T01:19:17.644 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.641+0000 7fa617010800 0 bdev(0x561d2e780700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2023-12-02T01:19:17.644 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.642+0000 7fa617010800 1 bdev(0x561d2e780700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:17.644 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.642+0000 7fa617010800 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-12-02T01:19:17.644 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.642+0000 7fa617010800 1 bluefs mount 2023-12-02T01:19:17.647 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.646+0000 7fa617010800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:17.657 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.656+0000 7fa617010800 1 bluefs mount shared_bdev_used = 0 2023-12-02T01:19:17.657 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.656+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.658+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.659+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:17.662 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.659+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:17.665 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.663+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:17.665 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.663+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2023-12-02T01:19:17.665 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.663+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2023-12-02T01:19:17.665 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.663+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-12-02T01:19:17.665 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.663+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2023-12-02T01:19:17.665 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.664+0000 7fa617010800 1 freelist init 2023-12-02T01:19:17.665 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.664+0000 7fa617010800 1 freelist _read_cfg 2023-12-02T01:19:17.666 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.664+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.72206e-07 2023-12-02T01:19:17.666 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.664+0000 7fa617010800 1 bluefs umount 2023-12-02T01:19:17.666 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.664+0000 7fa617010800 1 bdev(0x561d2e780700 td/osd-mapper/1/block) close 2023-12-02T01:19:17.906 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.905+0000 7fa617010800 1 bdev(0x561d2e780700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-12-02T01:19:17.907 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.905+0000 7fa617010800 0 bdev(0x561d2e780700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2023-12-02T01:19:17.907 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.905+0000 7fa617010800 1 bdev(0x561d2e780700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:17.907 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.905+0000 7fa617010800 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-12-02T01:19:17.907 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.905+0000 7fa617010800 1 bluefs mount 2023-12-02T01:19:17.914 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.912+0000 7fa617010800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:17.923 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.921+0000 7fa617010800 1 bluefs mount shared_bdev_used = 9568256 2023-12-02T01:19:17.923 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.921+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.923+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.923+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:17.927 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:17.927 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:17.927 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:17.927 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:17.927 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:17.927 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:17.924+0000 7fa617010800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:18.180 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:18.179+0000 7fa617010800 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 fe a9 55 c5 d6 7c 4a dd 91 8e |..X.....U..|J...| 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 30 3f ae 2a 3d e6 01 00 00 00 1c 00 00 00 00 00 |0?.*=...........| 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:* 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-12-02T01:19:18.182 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-12-02T01:19:18.183 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1c 00 00 00 0a 00 00 00 7d e2 71 6a 8c 41 |re........}.qj.A| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 40 21 8e aa 98 fc 79 eb cb 78 00 00 00 00 00 00 |@!....y..x......| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1c 00 00 00 d3 85 6a 65 c2 87 23 19 01 00 |........je..#...| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-12-02T01:19:18.184 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.185 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:18.186 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:18.187 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-12-02T01:19:18.188 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:18.189 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:18.179+0000 7fa617010800 1 bluefs umount 2023-12-02T01:19:18.189 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:18.179+0000 7fa617010800 1 bdev(0x561d2e780700 td/osd-mapper/1/block) close 2023-12-02T01:19:18.417 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:18.416+0000 7fa617010800 1 freelist shutdown 2023-12-02T01:19:18.417 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:18.416+0000 7fa617010800 1 bdev(0x561d2e780000 td/osd-mapper/1/block) close 2023-12-02T01:19:18.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-12-02T01:19:18.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-12-02T01:19:18.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-12-02T01:19:19.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:19.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:19.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-12-02T01:19:19.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-12-02T01:19:19.454 INFO:tasks.workunit.client.0.smithi162.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:19.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-12-02T01:19:19.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.09Kb6As0qn_the_val 2023-12-02T01:19:19.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:19.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.09Kb6As0qn_the_val' 2023-12-02T01:19:19.457 INFO:tasks.workunit.client.0.smithi162.stdout:Value dumped in: /tmp/tmp.09Kb6As0qn_the_val 2023-12-02T01:19:19.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.09Kb6As0qn_the_val 2023-12-02T01:19:20.091 INFO:tasks.workunit.client.0.smithi162.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-12-02T01:19:20.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:20.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.09Kb6As0qn_the_val 2023-12-02T01:19:20.500 INFO:tasks.workunit.client.0.smithi162.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-12-02T01:19:20.500 INFO:tasks.workunit.client.0.smithi162.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-12-02T01:19:20.500 INFO:tasks.workunit.client.0.smithi162.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-12-02T01:19:20.500 INFO:tasks.workunit.client.0.smithi162.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-12-02T01:19:20.500 INFO:tasks.workunit.client.0.smithi162.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-12-02T01:19:20.500 INFO:tasks.workunit.client.0.smithi162.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-12-02T01:19:20.501 INFO:tasks.workunit.client.0.smithi162.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-12-02T01:19:20.501 INFO:tasks.workunit.client.0.smithi162.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-12-02T01:19:20.501 INFO:tasks.workunit.client.0.smithi162.stdout:0000073 2023-12-02T01:19:20.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-12-02T01:19:20.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:21.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.09Kb6As0qn_the_val 2023-12-02T01:19:22.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:22.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:22.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2023-12-02T01:19:22.587 INFO:tasks.workunit.client.0.smithi162.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2023-12-02T01:19:22.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2023-12-02T01:19:22.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-12-02T01:19:22.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-12-02T01:19:22.613 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.611+0000 7f10bf441800 1 bdev(0x5624c4ef4000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-12-02T01:19:22.613 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.611+0000 7f10bf441800 0 bdev(0x5624c4ef4000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2023-12-02T01:19:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.612+0000 7f10bf441800 1 bdev(0x5624c4ef4000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.612+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2023-12-02T01:19:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.612+0000 7f10bf441800 1 bdev(0x5624c4ef4700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-12-02T01:19:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.612+0000 7f10bf441800 0 bdev(0x5624c4ef4700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2023-12-02T01:19:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.612+0000 7f10bf441800 1 bdev(0x5624c4ef4700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.612+0000 7f10bf441800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-12-02T01:19:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.612+0000 7f10bf441800 1 bluefs mount 2023-12-02T01:19:22.619 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.617+0000 7f10bf441800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:22.628 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.627+0000 7f10bf441800 1 bluefs mount shared_bdev_used = 0 2023-12-02T01:19:22.628 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.627+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:22.631 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.629+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:22.631 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.629+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:22.631 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.629+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:22.631 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.629+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.629+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.629+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.629+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.630+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.630+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.630+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.630+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.630+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.634+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.634+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-12-02T01:19:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.634+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-12-02T01:19:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.634+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-12-02T01:19:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.634+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-12-02T01:19:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.635+0000 7f10bf441800 1 freelist init 2023-12-02T01:19:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.635+0000 7f10bf441800 1 freelist _read_cfg 2023-12-02T01:19:22.637 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.635+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.34059e-07 2023-12-02T01:19:22.637 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.635+0000 7f10bf441800 1 bluefs umount 2023-12-02T01:19:22.637 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.635+0000 7f10bf441800 1 bdev(0x5624c4ef4700 td/osd-mapper/2/block) close 2023-12-02T01:19:22.875 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.874+0000 7f10bf441800 1 bdev(0x5624c4ef4700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-12-02T01:19:22.875 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.874+0000 7f10bf441800 0 bdev(0x5624c4ef4700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2023-12-02T01:19:22.876 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.874+0000 7f10bf441800 1 bdev(0x5624c4ef4700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-12-02T01:19:22.876 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.874+0000 7f10bf441800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-12-02T01:19:22.876 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.874+0000 7f10bf441800 1 bluefs mount 2023-12-02T01:19:22.885 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.884+0000 7f10bf441800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-12-02T01:19:22.894 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.893+0000 7f10bf441800 1 bluefs mount shared_bdev_used = 9568256 2023-12-02T01:19:22.894 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.893+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-12-02T01:19:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-12-02T01:19:22.898 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-12-02T01:19:22.898 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-12-02T01:19:22.898 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:22.895+0000 7f10bf441800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-12-02T01:19:23.112 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:23.111+0000 7f10bf441800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 fe a9 55 c5 d6 7c 4a dd 91 8e |..X.....U..|J...| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 30 3f ae 2a 3d e6 02 00 00 00 1c 00 00 00 00 00 |0?.*=...........| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:* 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-12-02T01:19:23.114 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-12-02T01:19:23.115 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 be d8 8b 9b 51 7a |re............Qz| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 45 60 b6 51 52 06 fe 99 2d 0b 00 00 00 00 00 00 |E`.QR...-.......| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1c 00 00 00 f4 85 6a 65 23 74 78 3a 01 00 |........je#tx:..| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-12-02T01:19:23.116 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.117 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:23.118 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:23.119 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-12-02T01:19:23.120 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:23.121 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:23.111+0000 7f10bf441800 1 bluefs umount 2023-12-02T01:19:23.121 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:23.112+0000 7f10bf441800 1 bdev(0x5624c4ef4700 td/osd-mapper/2/block) close 2023-12-02T01:19:23.349 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:23.348+0000 7f10bf441800 1 freelist shutdown 2023-12-02T01:19:23.349 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:23.348+0000 7f10bf441800 1 bdev(0x5624c4ef4000 td/osd-mapper/2/block) close 2023-12-02T01:19:23.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-12-02T01:19:23.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-12-02T01:19:23.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-12-02T01:19:24.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:24.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:24.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-12-02T01:19:24.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-12-02T01:19:24.428 INFO:tasks.workunit.client.0.smithi162.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:24.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-12-02T01:19:24.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.AVIq65Czaa_the_val 2023-12-02T01:19:24.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:24.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.AVIq65Czaa_the_val' 2023-12-02T01:19:24.430 INFO:tasks.workunit.client.0.smithi162.stdout:Value dumped in: /tmp/tmp.AVIq65Czaa_the_val 2023-12-02T01:19:24.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.AVIq65Czaa_the_val 2023-12-02T01:19:24.923 INFO:tasks.workunit.client.0.smithi162.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-12-02T01:19:25.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:25.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.AVIq65Czaa_the_val 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-12-02T01:19:25.223 INFO:tasks.workunit.client.0.smithi162.stdout:0000073 2023-12-02T01:19:25.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-12-02T01:19:25.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:26.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.AVIq65Czaa_the_val 2023-12-02T01:19:27.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:27.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 ' 2023-12-02T01:19:27.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 2023-12-02T01:19:27.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-12-02T01:19:27.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:27.309 INFO:tasks.workunit.client.0.smithi162.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ 2023-12-02T01:19:27.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/' 2023-12-02T01:19:27.309 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2023-12-02T01:19:27.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2023-12-02T01:19:27.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:19:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80181/$cluster-$name.asok' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80181/$cluster-$name.asok' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-12-02T01:19:27.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:19:27.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:19:27.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:19:27.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:19:27.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-12-02T01:19:27.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-12-02T01:19:27.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2023-12-02T01:19:27.316 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2023-12-02T01:19:27.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-12-02T01:19:27.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.80181/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:19:27.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2023-12-02T01:19:27.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-12-02T01:19:27.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-12-02T01:19:27.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-12-02T01:19:27.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-12-02T01:19:27.352 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:27.350+0000 7f000aeab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:27.354 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:27.353+0000 7f000aeab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:27.356 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:27.354+0000 7f000aeab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:27.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-12-02T01:19:27.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:19:27.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-12-02T01:19:27.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:19:27.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:19:27.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:27.604 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:19:27.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:19:27.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:27.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:19:27.871 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:27.870+0000 7f000aeab500 -1 Falling back to public interface 2023-12-02T01:19:27.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:28.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:28.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:28.890 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:19:28.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:19:28.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:28.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:19:29.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:30.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:30.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:30.171 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:19:30.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:19:30.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:30.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:19:30.198 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:30.196+0000 7f000aeab500 -1 osd.0 28 log_to_monitors true 2023-12-02T01:19:30.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:31.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:31.537 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:19:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:19:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:19:31.815 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 31 up_thru 31 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/2390319851,v1:127.0.0.1:6803/2390319851] [v2:127.0.0.1:6804/2390319851,v1:127.0.0.1:6805/2390319851] exists,up 819f62a4-8f8b-48cd-90ce-ba9429c124f8 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-12-02T01:19:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:19:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80181/$cluster-$name.asok' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80181/$cluster-$name.asok' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2023-12-02T01:19:31.821 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2023-12-02T01:19:31.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-12-02T01:19:31.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.80181/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:19:31.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2023-12-02T01:19:31.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-12-02T01:19:31.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-12-02T01:19:31.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-12-02T01:19:31.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-12-02T01:19:31.851 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:31.850+0000 7fbc006c7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:31.878 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:31.877+0000 7fbc006c7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:31.879 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:31.878+0000 7fbc006c7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:32.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-12-02T01:19:32.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:19:32.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-12-02T01:19:32.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:19:32.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:19:32.120 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:19:32.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:32.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:19:32.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:32.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:19:32.400 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:32.399+0000 7fbc006c7500 -1 Falling back to public interface 2023-12-02T01:19:32.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:33.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:33.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:33.435 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:19:33.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:19:33.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:33.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:19:33.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:34.711 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:34.710+0000 7fbc006c7500 -1 osd.1 28 log_to_monitors true 2023-12-02T01:19:34.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:34.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:34.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:19:34.722 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:19:34.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:34.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:19:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:36.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:36.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:36.057 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:19:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:19:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:19:36.353 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/638112112,v1:127.0.0.1:6811/638112112] [v2:127.0.0.1:6812/638112112,v1:127.0.0.1:6813/638112112] exists,up 7de2716a-8c41-4021-8eaa-98fc79ebcb78 2023-12-02T01:19:36.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:19:36.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:19:36.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:19:36.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-12-02T01:19:36.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80181/$cluster-$name.asok' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80181/$cluster-$name.asok' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:19:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:19:36.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:19:36.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:19:36.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-12-02T01:19:36.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-12-02T01:19:36.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2023-12-02T01:19:36.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-12-02T01:19:36.360 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2023-12-02T01:19:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=fea955c5-d67c-4add-918e-303fae2a3de6 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.80181/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:19:36.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2023-12-02T01:19:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-12-02T01:19:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-12-02T01:19:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-12-02T01:19:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-12-02T01:19:36.389 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:36.388+0000 7fb5e3d4a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:36.391 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:36.390+0000 7fb5e3d4a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:36.392 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:36.391+0000 7fb5e3d4a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:19:36.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-12-02T01:19:36.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:19:36.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-12-02T01:19:36.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:19:36.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:19:36.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:36.636 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:19:36.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:19:36.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:36.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:19:36.907 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:36.906+0000 7fb5e3d4a500 -1 Falling back to public interface 2023-12-02T01:19:36.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:37.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:37.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:37.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:19:37.933 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:19:37.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:37.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:19:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:39.102 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:19:39.100+0000 7fb5e3d4a500 -1 osd.2 28 log_to_monitors true 2023-12-02T01:19:39.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:39.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:39.210 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:19:39.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:19:39.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:39.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:19:39.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:19:40.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:19:40.491 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:19:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:19:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:19:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:19:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:19:40.858 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 38 up_thru 38 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/508913675,v1:127.0.0.1:6819/508913675] [v2:127.0.0.1:6820/508913675,v1:127.0.0.1:6821/508913675] exists,up bed88b9b-517a-4560-b651-5206fe992d0b 2023-12-02T01:19:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:19:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:19:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:19:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2023-12-02T01:19:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2023-12-02T01:19:41.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2023-12-02T01:19:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2023-12-02T01:19:41.960 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:19:41.960 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "18.0.0-7631-gbed76814", 2023-12-02T01:19:41.961 INFO:tasks.workunit.client.0.smithi162.stdout: "release": "reef", 2023-12-02T01:19:41.961 INFO:tasks.workunit.client.0.smithi162.stdout: "release_type": "dev" 2023-12-02T01:19:41.961 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:19:41.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:41.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2023-12-02T01:19:42.065 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:19:42.065 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "18.0.0-7631-gbed76814", 2023-12-02T01:19:42.065 INFO:tasks.workunit.client.0.smithi162.stdout: "release": "reef", 2023-12-02T01:19:42.065 INFO:tasks.workunit.client.0.smithi162.stdout: "release_type": "dev" 2023-12-02T01:19:42.065 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:19:42.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-12-02T01:19:42.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2023-12-02T01:19:42.170 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:19:42.170 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "18.0.0-7631-gbed76814", 2023-12-02T01:19:42.170 INFO:tasks.workunit.client.0.smithi162.stdout: "release": "reef", 2023-12-02T01:19:42.170 INFO:tasks.workunit.client.0.smithi162.stdout: "release_type": "dev" 2023-12-02T01:19:42.170 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:19:42.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "objxxx", 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: "clones": [ 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 1, 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap01" 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.206 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap02" 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap13" 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2023-12-02T01:19:42.207 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 4, 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap24" 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap25" 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2023-12-02T01:19:42.208 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stdout: { 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stdout: "id": "head", 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [], 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stdout: } 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:19:42.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2023-12-02T01:19:42.573 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is unset 2023-12-02T01:19:42.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2023-12-02T01:19:42.949 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is unset 2023-12-02T01:19:42.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-12-02T01:19:42.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2023-12-02T01:19:43.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2023-12-02T01:19:43.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2023-12-02T01:19:43.513 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:19:43.513 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 peering 2023-12-02T01:19:39.698458+0000 26'7 37:46 [1,2,0] 1 [1,2,0] 1 26'7 2023-12-02T01:18:56.951407+0000 26'7 2023-12-02T01:18:56.951407+0000 0 1 no scrub is scheduled 4 0 2023-12-02T01:19:43.513 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:19:40.226410+0000 0'0 39:37 [0,1,2] 0 [0,1,2] 0 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:43.513 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2023-12-02T01:19:35.318778+0000 0'0 34:10 [0,1] 0 [0,1] 0 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 no scrub is scheduled 0 0 2023-12-02T01:19:43.513 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2023-12-02T01:19:39.698346+0000 0'0 37:21 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 no scrub is scheduled 0 0 2023-12-02T01:19:43.513 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:43.513 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:19:43.514 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:19:43.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2023-12-02T01:19:45.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2023-12-02T01:19:45.630 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:19:45.631 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2023-12-02T01:19:45.631 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:19:45.631 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:19:45.631 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:19:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2023-12-02T01:19:50.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2023-12-02T01:19:50.903 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:19:50.903 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2023-12-02T01:19:40.763421+0000 26'7 42:59 [1,2,0] 1 [1,2,0] 1 26'7 2023-12-02T01:18:56.951407+0000 26'7 2023-12-02T01:18:56.951407+0000 0 1 periodic scrub scheduled @ 2023-12-03T01:18:56.951407+0000 4 0 2023-12-02T01:19:50.903 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:19:40.226410+0000 0'0 42:39 [0,1,2] 0 [0,1,2] 0 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:50.903 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:19:40.142849+0000 0'0 41:25 [2,0,1] 2 [2,0,1] 2 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:50.903 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:19:40.720744+0000 0'0 42:33 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:50.903 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:50.903 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:19:50.904 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:19:50.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:50.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-12-02T01:19:50.915 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:19:45.886+0000 7fbbe2e35640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-12-02T01:19:50.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2023-12-02T01:19:50.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-12-02T01:19:50.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2023-12-02T01:19:50.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2023-12-02T01:19:50.920 INFO:tasks.workunit.client.0.smithi162.stdout:prev count: 1 2023-12-02T01:19:50.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2023-12-02T01:19:50.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2023-12-02T01:19:51.013 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:19:51.013 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2023-12-02T01:19:51.013 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:19:51.013 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:19:51.013 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:19:51.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2023-12-02T01:19:56.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2023-12-02T01:19:56.282 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:19:56.282 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2023-12-02T01:19:51.842231+0000 26'7 42:77 [1,2,0] 1 [1,2,0] 1 26'7 2023-12-02T01:19:51.842150+0000 26'7 2023-12-02T01:19:51.842150+0000 0 1 periodic scrub scheduled @ 2023-12-03T01:19:51.842150+0000 4 0 2023-12-02T01:19:56.283 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:19:40.226410+0000 0'0 42:39 [0,1,2] 0 [0,1,2] 0 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:56.283 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:19:40.142849+0000 0'0 41:25 [2,0,1] 2 [2,0,1] 2 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:56.283 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:19:40.720744+0000 0'0 42:33 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:18:46.794544+0000 0'0 2023-12-02T01:18:46.794544+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:18:46.794544+0000 0 0 2023-12-02T01:19:56.283 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:19:56.283 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:19:56.283 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:19:56.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:56.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-12-02T01:19:56.295 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:19:45.886+0000 7fbbe2e35640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-12-02T01:19:56.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-12-02T01:19:56.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2023-12-02T01:19:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2023-12-02T01:19:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-12-02T01:19:56.298 INFO:tasks.workunit.client.0.smithi162.stdout:current count: 1 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:19:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:19:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:19:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-12-02T01:19:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-12-02T01:19:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-12-02T01:19:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2023-12-02T01:19:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-12-02T01:19:57.891 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:57.891 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-12-02T01:19:57.891 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-12-02T01:19:57.891 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-12-02T01:19:57.891 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:57.891 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-12-02T01:19:57.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-12-02T01:19:57.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-12-02T01:19:57.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-12-02T01:19:57.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2023-12-02T01:19:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:19:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:19:58.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:19:58.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:19:58.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:19:58.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:19:58.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:19:58.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:19:58.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:19:58.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:19:58.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:19:58.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:19:58.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:19:58.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:19:58.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:19:58.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:19:58.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-12-02T01:19:58.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-12-02T01:19:58.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:19:58.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:58.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:19:58.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80181 2023-12-02T01:19:58.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:19:58.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:19:58.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:19:58.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:19:58.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:19:58.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:19:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:19:58.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:19:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:19:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:19:58.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:19:58.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:19:58.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:19:58.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:19:58.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:19:58.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:19:58.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-12-02T01:19:58.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-12-02T01:19:58.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:19:58.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:58.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80181 2023-12-02T01:19:58.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80181 2023-12-02T01:19:58.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:19:58.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:19:58.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2023-12-02T01:19:58.834 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-02T01:19:58.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-02T01:19:58.903 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2023-12-02T01:19:58.903 DEBUG:teuthology.orchestra.run.smithi162:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2023-12-02T01:19:58.966 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2023-12-02T01:19:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2023-12-02T01:19:58.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2023-12-02T01:19:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none ' 2023-12-02T01:19:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2023-12-02T01:19:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2023-12-02T01:19:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-12-02T01:19:58.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2023-12-02T01:19:58.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_recovery_scrub_2' 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.96539 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2023-12-02T01:19:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:19:58.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:19:58.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:19:58.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:19:58.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:19:58.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:19:58.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:19:58.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:19:58.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:19:58.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:19:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:19:58.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:19:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:19:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-12-02T01:19:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-12-02T01:19:58.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:19:58.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:58.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:19:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96539 2023-12-02T01:19:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:19:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:19:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-12-02T01:19:58.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-12-02T01:19:58.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:58.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:19:58.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96539 2023-12-02T01:19:58.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-12-02T01:19:58.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-12-02T01:19:58.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-12-02T01:19:58.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-12-02T01:19:59.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-12-02T01:19:59.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:19:59.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:19:59.390 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:19:59.390 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:59.390 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:19:59.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:19:59.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-12-02T01:19:59.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-12-02T01:19:59.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-12-02T01:19:59.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-12-02T01:19:59.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-12-02T01:19:59.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-12-02T01:19:59.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-12-02T01:19:59.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-12-02T01:19:59.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-12-02T01:19:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-12-02T01:19:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:19:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:19:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96539/ceph-mon.a.asok 2023-12-02T01:19:59.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-12-02T01:19:59.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96539/ceph-mon.a.asok config get fsid 2023-12-02T01:19:59.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-12-02T01:19:59.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-12-02T01:19:59.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-12-02T01:19:59.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-12-02T01:19:59.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-12-02T01:19:59.667 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96539/ceph-mon.a.asok 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-12-02T01:19:59.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96539/ceph-mon.a.asok config get mon_host 2023-12-02T01:19:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2023-12-02T01:19:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-12-02T01:19:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-12-02T01:19:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-12-02T01:19:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-12-02T01:19:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-12-02T01:19:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-12-02T01:20:00.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-12-02T01:20:00.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:20:00.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:20:00.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:20:00.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:20:00.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:20:00.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:00.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-12-02T01:20:00.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-12-02T01:20:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2023-12-02T01:20:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2023-12-02T01:20:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2023-12-02T01:20:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2023-12-02T01:20:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2023-12-02T01:20:00.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-12-02T01:20:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:20:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:20:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:20:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:20:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:20:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-12-02T01:20:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-12-02T01:20:00.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:20:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e7c67da-bffe-4fb0-8b82-e129705a5ddf 2023-12-02T01:20:00.603 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 3e7c67da-bffe-4fb0-8b82-e129705a5ddf 2023-12-02T01:20:00.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3e7c67da-bffe-4fb0-8b82-e129705a5ddf' 2023-12-02T01:20:00.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:20:00.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAhmpllcC/JBAA+81QsrvFOweUfufcQ9LWfg== 2023-12-02T01:20:00.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAhmpllcC/JBAA+81QsrvFOweUfufcQ9LWfg=="}' 2023-12-02T01:20:00.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e7c67da-bffe-4fb0-8b82-e129705a5ddf -i td/osd-recovery-scrub/0/new.json 2023-12-02T01:20:00.823 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:20:00.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-12-02T01:20:00.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBAhmpllcC/JBAA+81QsrvFOweUfufcQ9LWfg== --osd-uuid 3e7c67da-bffe-4fb0-8b82-e129705a5ddf 2023-12-02T01:20:00.862 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:00.860+0000 7f9a2169a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:00.873 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:00.872+0000 7f9a2169a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:00.874 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:00.873+0000 7f9a2169a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:00.875 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:00.874+0000 7f9a2169a500 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-12-02T01:20:00.875 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:00.874+0000 7f9a2169a500 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-12-02T01:20:00.875 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:00.874+0000 7f9a2169a500 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-12-02T01:20:00.957 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:00.956+0000 7f9a2169a500 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-12-02T01:20:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-12-02T01:20:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:20:06.484 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2023-12-02T01:20:06.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-12-02T01:20:06.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:20:06.835 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2023-12-02T01:20:06.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-12-02T01:20:06.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-12-02T01:20:06.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:20:06.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:20:06.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:20:06.861 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:06.860+0000 7fd90601b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:06.863 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:06.862+0000 7fd90601b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:06.864 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:06.863+0000 7fd90601b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:07.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-12-02T01:20:07.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:20:07.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-12-02T01:20:07.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:20:07.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:20:07.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:07.106 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:20:07.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:20:07.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:07.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:20:07.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:07.390 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:07.389+0000 7fd90601b500 -1 Falling back to public interface 2023-12-02T01:20:08.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:08.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:08.377 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:20:08.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:20:08.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:08.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:20:08.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:09.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:09.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:09.653 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:20:09.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:20:09.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:09.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:20:09.737 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:09.735+0000 7fd90601b500 -1 osd.0 0 log_to_monitors true 2023-12-02T01:20:09.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:10.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:10.951 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:20:10.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:10.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:20:10.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:10.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:20:11.257 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3857028544,v1:127.0.0.1:6803/3857028544] [v2:127.0.0.1:6804/3857028544,v1:127.0.0.1:6805/3857028544] exists,up 3e7c67da-bffe-4fb0-8b82-e129705a5ddf 2023-12-02T01:20:11.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:20:11.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:20:11.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:20:11.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:20:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:20:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5a11124-1a2c-4698-89a0-fe4ee7201f4c 2023-12-02T01:20:11.263 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 f5a11124-1a2c-4698-89a0-fe4ee7201f4c 2023-12-02T01:20:11.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f5a11124-1a2c-4698-89a0-fe4ee7201f4c' 2023-12-02T01:20:11.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:20:11.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLhmplPeeJEBAA7YHukJw1oSvFOPJw+WE8SQ== 2023-12-02T01:20:11.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLhmplPeeJEBAA7YHukJw1oSvFOPJw+WE8SQ=="}' 2023-12-02T01:20:11.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5a11124-1a2c-4698-89a0-fe4ee7201f4c -i td/osd-recovery-scrub/1/new.json 2023-12-02T01:20:11.623 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:20:11.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-12-02T01:20:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBLhmplPeeJEBAA7YHukJw1oSvFOPJw+WE8SQ== --osd-uuid f5a11124-1a2c-4698-89a0-fe4ee7201f4c 2023-12-02T01:20:11.663 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:11.661+0000 7fb034d72500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:11.710 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:11.709+0000 7fb034d72500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:11.711 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:11.710+0000 7fb034d72500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:11.712 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:11.710+0000 7fb034d72500 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-12-02T01:20:11.712 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:11.710+0000 7fb034d72500 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-12-02T01:20:11.712 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:11.710+0000 7fb034d72500 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-12-02T01:20:11.782 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:11.781+0000 7fb034d72500 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-12-02T01:20:17.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-12-02T01:20:17.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:20:17.606 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2023-12-02T01:20:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-12-02T01:20:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:20:17.952 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2023-12-02T01:20:17.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-12-02T01:20:17.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-12-02T01:20:17.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:20:17.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:20:17.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:20:17.979 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:17.977+0000 7fe1f8412500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:17.981 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:17.979+0000 7fe1f8412500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:17.982 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:17.980+0000 7fe1f8412500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:18.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-12-02T01:20:18.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:20:18.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-12-02T01:20:18.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:20:18.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:20:18.222 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:20:18.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:18.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:20:18.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:18.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:20:18.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:18.520 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:18.519+0000 7fe1f8412500 -1 Falling back to public interface 2023-12-02T01:20:19.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:19.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:19.490 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:20:19.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:20:19.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:19.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:20:19.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:20.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:20.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:20.765 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:20:20.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:20:20.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:20.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:20:20.994 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:20.993+0000 7fe1f8412500 -1 osd.1 0 log_to_monitors true 2023-12-02T01:20:21.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:22.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:22.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:22.044 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:20:22.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:20:22.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:22.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:20:22.398 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2471862270,v1:127.0.0.1:6811/2471862270] [v2:127.0.0.1:6812/2471862270,v1:127.0.0.1:6813/2471862270] exists,up f5a11124-1a2c-4698-89a0-fe4ee7201f4c 2023-12-02T01:20:22.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:20:22.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:20:22.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:20:22.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:20:22.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3f1f636-f26e-4288-9a2a-e9c25f58cd34 2023-12-02T01:20:22.405 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 f3f1f636-f26e-4288-9a2a-e9c25f58cd34 2023-12-02T01:20:22.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f3f1f636-f26e-4288-9a2a-e9c25f58cd34' 2023-12-02T01:20:22.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:20:22.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWhmplKyEHGRAAnKblSkFxYMnJgDp5Kw9A5w== 2023-12-02T01:20:22.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWhmplKyEHGRAAnKblSkFxYMnJgDp5Kw9A5w=="}' 2023-12-02T01:20:22.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3f1f636-f26e-4288-9a2a-e9c25f58cd34 -i td/osd-recovery-scrub/2/new.json 2023-12-02T01:20:22.738 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:20:22.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-12-02T01:20:22.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBWhmplKyEHGRAAnKblSkFxYMnJgDp5Kw9A5w== --osd-uuid f3f1f636-f26e-4288-9a2a-e9c25f58cd34 2023-12-02T01:20:22.779 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:22.777+0000 7f14bcfab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:22.814 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:22.813+0000 7f14bcfab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:22.815 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:22.814+0000 7f14bcfab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:22.816 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:22.814+0000 7f14bcfab500 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-12-02T01:20:22.816 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:22.815+0000 7f14bcfab500 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-12-02T01:20:22.816 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:22.815+0000 7f14bcfab500 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-12-02T01:20:22.881 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:22.880+0000 7f14bcfab500 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-12-02T01:20:28.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-12-02T01:20:28.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:20:28.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-12-02T01:20:28.812 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2023-12-02T01:20:28.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:20:29.156 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2023-12-02T01:20:29.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-12-02T01:20:29.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-12-02T01:20:29.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:20:29.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:20:29.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:20:29.183 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:29.181+0000 7f76bbc12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:29.185 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:29.184+0000 7f76bbc12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:29.186 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:29.185+0000 7f76bbc12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-12-02T01:20:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:20:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-12-02T01:20:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:20:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:20:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:29.427 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:20:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:20:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:20:29.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:29.709 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:29.707+0000 7f76bbc12500 -1 Falling back to public interface 2023-12-02T01:20:30.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:30.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:30.693 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:20:30.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:20:30.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:30.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:20:30.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:31.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:31.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:31.960 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:20:31.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:20:31.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:31.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:20:32.020 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:32.019+0000 7f76bbc12500 -1 osd.2 0 log_to_monitors true 2023-12-02T01:20:32.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:33.230 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:20:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:20:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:20:33.528 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3170692229,v1:127.0.0.1:6819/3170692229] [v2:127.0.0.1:6820/3170692229,v1:127.0.0.1:6821/3170692229] exists,up f3f1f636-f26e-4288-9a2a-e9c25f58cd34 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-12-02T01:20:33.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:20:33.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:20:33.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39fd2a01-844f-4d07-b5f5-683d620da12f 2023-12-02T01:20:33.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 39fd2a01-844f-4d07-b5f5-683d620da12f' 2023-12-02T01:20:33.535 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 39fd2a01-844f-4d07-b5f5-683d620da12f 2023-12-02T01:20:33.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:20:33.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhhmplqT3DIBAAbHz1/taxjugwr8+JIDTzvg== 2023-12-02T01:20:33.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhhmplqT3DIBAAbHz1/taxjugwr8+JIDTzvg=="}' 2023-12-02T01:20:33.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39fd2a01-844f-4d07-b5f5-683d620da12f -i td/osd-recovery-scrub/3/new.json 2023-12-02T01:20:33.906 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:20:33.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-12-02T01:20:33.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBhhmplqT3DIBAAbHz1/taxjugwr8+JIDTzvg== --osd-uuid 39fd2a01-844f-4d07-b5f5-683d620da12f 2023-12-02T01:20:33.946 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:33.944+0000 7f994ef2b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:33.981+0000 7f994ef2b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:33.982+0000 7f994ef2b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:33.983+0000 7f994ef2b500 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-12-02T01:20:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:33.983+0000 7f994ef2b500 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-12-02T01:20:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:33.983+0000 7f994ef2b500 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-12-02T01:20:34.049 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:34.048+0000 7f994ef2b500 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-12-02T01:20:39.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-12-02T01:20:39.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:20:39.880 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2023-12-02T01:20:39.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-12-02T01:20:39.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:20:40.224 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2023-12-02T01:20:40.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-12-02T01:20:40.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-12-02T01:20:40.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:20:40.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:20:40.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:20:40.250 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:40.249+0000 7f85f7c0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:40.252 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:40.251+0000 7f85f7c0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:40.253 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:40.252+0000 7f85f7c0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:20:40.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-12-02T01:20:40.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:20:40.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-12-02T01:20:40.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:20:40.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:20:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:20:40.492 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:20:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:20:40.767 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:40.766+0000 7f85f7c0c500 -1 Falling back to public interface 2023-12-02T01:20:40.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:41.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:41.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:41.782 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:20:41.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:20:41.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:41.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:20:42.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:43.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:43.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:43.053 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:20:43.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:20:43.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:43.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:20:43.174 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:20:43.170+0000 7f85f7c0c500 -1 osd.3 0 log_to_monitors true 2023-12-02T01:20:43.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:20:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:20:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:20:44.381 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:20:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:20:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:20:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:20:44.687 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3752461858,v1:127.0.0.1:6827/3752461858] [v2:127.0.0.1:6828/3752461858,v1:127.0.0.1:6829/3752461858] exists,up 39fd2a01-844f-4d07-b5f5-683d620da12f 2023-12-02T01:20:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:20:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:20:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:20:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2023-12-02T01:20:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2023-12-02T01:20:45.103 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test' created 2023-12-02T01:20:45.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-12-02T01:20:46.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2023-12-02T01:20:46.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-12-02T01:20:46.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-12-02T01:20:46.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-12-02T01:20:46.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-12-02T01:20:46.117 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-12-02T01:20:46.117 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-12-02T01:20:46.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-12-02T01:20:46.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-12-02T01:20:46.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-12-02T01:20:46.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-12-02T01:20:46.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-12-02T01:20:46.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-12-02T01:20:46.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-12-02T01:20:46.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2023-12-02T01:20:46.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2023-12-02T01:20:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2023-12-02T01:20:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:1 2023-12-02T01:20:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:2 2023-12-02T01:20:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2023-12-02T01:20:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2023-12-02T01:20:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:20:46.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-12-02T01:20:46.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2023-12-02T01:20:46.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2023-12-02T01:20:46.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2023-12-02T01:20:46.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:20:46.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-12-02T01:20:46.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2023-12-02T01:20:46.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2023-12-02T01:20:46.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2023-12-02T01:20:46.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:20:46.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-12-02T01:20:46.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2023-12-02T01:20:46.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2023-12-02T01:20:46.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2023-12-02T01:20:46.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:20:46.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-12-02T01:20:46.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2023-12-02T01:20:46.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2023-12-02T01:20:46.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444 3-85899345922' 2023-12-02T01:20:46.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:20:46.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2023-12-02T01:20:46.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:20:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2023-12-02T01:20:46.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2023-12-02T01:20:46.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:20:46.979 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836489 2023-12-02T01:20:46.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2023-12-02T01:20:46.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2023-12-02T01:20:46.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2023-12-02T01:20:47.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2023-12-02T01:20:47.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2023-12-02T01:20:48.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2023-12-02T01:20:48.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2023-12-02T01:20:48.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2023-12-02T01:20:48.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2023-12-02T01:20:49.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2023-12-02T01:20:49.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2023-12-02T01:20:49.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2023-12-02T01:20:49.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:20:49.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2023-12-02T01:20:49.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:20:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2023-12-02T01:20:49.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2023-12-02T01:20:49.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:20:49.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2023-12-02T01:20:49.827 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672966 2023-12-02T01:20:49.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2023-12-02T01:20:49.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2023-12-02T01:20:50.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2023-12-02T01:20:50.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:20:50.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2023-12-02T01:20:50.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:20:50.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2023-12-02T01:20:50.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2023-12-02T01:20:50.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:20:50.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2023-12-02T01:20:50.100 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509444 2023-12-02T01:20:50.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2023-12-02T01:20:50.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2023-12-02T01:20:50.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2023-12-02T01:20:50.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:20:50.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2023-12-02T01:20:50.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:20:50.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2023-12-02T01:20:50.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2023-12-02T01:20:50.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:20:50.369 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345922 2023-12-02T01:20:50.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2023-12-02T01:20:50.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2023-12-02T01:20:50.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2023-12-02T01:20:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345922 2023-12-02T01:20:50.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-12-02T01:20:50.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-12-02T01:20:50.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-12-02T01:20:51.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-12-02T01:20:51.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-12-02T01:20:51.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-12-02T01:20:51.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-12-02T01:20:51.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-12-02T01:20:51.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-12-02T01:20:51.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-12-02T01:20:51.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-12-02T01:20:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-12-02T01:20:51.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-12-02T01:20:51.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-12-02T01:20:51.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-12-02T01:20:51.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-12-02T01:20:51.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-12-02T01:20:51.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-12-02T01:20:51.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2023-12-02T01:20:51.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2023-12-02T01:20:51.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2023-12-02T01:20:51.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2023-12-02T01:20:51.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2023-12-02T01:20:52.126 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:20:52.126 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:20:46.482197+0000 0'0 23:8 [1] 1 [1] 1 0'0 2023-12-02T01:20:44.976941+0000 0'0 2023-12-02T01:20:44.976941+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:04:06.576645+0000 0 0 2023-12-02T01:20:52.126 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:20:52.126 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:20:52.127 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:20:52.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.96539 bs=1M count=50 2023-12-02T01:20:52.277 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records in 2023-12-02T01:20:52.277 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records out 2023-12-02T01:20:52.277 INFO:tasks.workunit.client.0.smithi162.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.139725 s, 375 MB/s 2023-12-02T01:20:52.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2023-12-02T01:20:52.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:20:52.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.96539 2023-12-02T01:20:53.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:20:53.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.96539 2023-12-02T01:20:55.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:20:55.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.96539 2023-12-02T01:20:56.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:20:56.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.96539 2023-12-02T01:20:58.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:20:58.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.96539 2023-12-02T01:20:59.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:20:59.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.96539 2023-12-02T01:21:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.96539 2023-12-02T01:21:02.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:02.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.96539 2023-12-02T01:21:04.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:04.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.96539 2023-12-02T01:21:05.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:05.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.96539 2023-12-02T01:21:07.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:07.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.96539 2023-12-02T01:21:08.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:08.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.96539 2023-12-02T01:21:10.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:10.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.96539 2023-12-02T01:21:11.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:11.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.96539 2023-12-02T01:21:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.96539 2023-12-02T01:21:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:14.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.96539 2023-12-02T01:21:16.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:16.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.96539 2023-12-02T01:21:17.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:17.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.96539 2023-12-02T01:21:19.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:19.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.96539 2023-12-02T01:21:20.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:20.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.96539 2023-12-02T01:21:22.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:22.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.96539 2023-12-02T01:21:23.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:23.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.96539 2023-12-02T01:21:25.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:25.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.96539 2023-12-02T01:21:26.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:26.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.96539 2023-12-02T01:21:28.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:28.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.96539 2023-12-02T01:21:29.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:29.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.96539 2023-12-02T01:21:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.96539 2023-12-02T01:21:33.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:33.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.96539 2023-12-02T01:21:34.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:34.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.96539 2023-12-02T01:21:35.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:35.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.96539 2023-12-02T01:21:37.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:37.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.96539 2023-12-02T01:21:38.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:38.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.96539 2023-12-02T01:21:40.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:40.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.96539 2023-12-02T01:21:41.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:41.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.96539 2023-12-02T01:21:43.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:43.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.96539 2023-12-02T01:21:44.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:44.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.96539 2023-12-02T01:21:46.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:46.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.96539 2023-12-02T01:21:47.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:47.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.96539 2023-12-02T01:21:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.96539 2023-12-02T01:21:50.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:50.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.96539 2023-12-02T01:21:52.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:52.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.96539 2023-12-02T01:21:53.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:53.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.96539 2023-12-02T01:21:55.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:55.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.96539 2023-12-02T01:21:56.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:56.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.96539 2023-12-02T01:21:58.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:58.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.96539 2023-12-02T01:21:59.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:21:59.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.96539 2023-12-02T01:22:01.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:01.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.96539 2023-12-02T01:22:02.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:02.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.96539 2023-12-02T01:22:04.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:04.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.96539 2023-12-02T01:22:05.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:05.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.96539 2023-12-02T01:22:07.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:07.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.96539 2023-12-02T01:22:08.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:08.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.96539 2023-12-02T01:22:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.96539 2023-12-02T01:22:11.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:11.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.96539 2023-12-02T01:22:13.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:13.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.96539 2023-12-02T01:22:14.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:14.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.96539 2023-12-02T01:22:16.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:16.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.96539 2023-12-02T01:22:17.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:17.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.96539 2023-12-02T01:22:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.96539 2023-12-02T01:22:20.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:20.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.96539 2023-12-02T01:22:22.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:22.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.96539 2023-12-02T01:22:23.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:23.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.96539 2023-12-02T01:22:25.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:25.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.96539 2023-12-02T01:22:26.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:26.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.96539 2023-12-02T01:22:28.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:28.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.96539 2023-12-02T01:22:29.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:29.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.96539 2023-12-02T01:22:31.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:31.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.96539 2023-12-02T01:22:32.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:32.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.96539 2023-12-02T01:22:34.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:34.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.96539 2023-12-02T01:22:35.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:35.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.96539 2023-12-02T01:22:37.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:37.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.96539 2023-12-02T01:22:38.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:38.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.96539 2023-12-02T01:22:40.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:40.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.96539 2023-12-02T01:22:41.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:41.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.96539 2023-12-02T01:22:43.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:43.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.96539 2023-12-02T01:22:44.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:44.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.96539 2023-12-02T01:22:46.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:46.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.96539 2023-12-02T01:22:47.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:47.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.96539 2023-12-02T01:22:49.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:49.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.96539 2023-12-02T01:22:50.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:50.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.96539 2023-12-02T01:22:52.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:52.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.96539 2023-12-02T01:22:53.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:53.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.96539 2023-12-02T01:22:55.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:55.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.96539 2023-12-02T01:22:57.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:57.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.96539 2023-12-02T01:22:58.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:58.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.96539 2023-12-02T01:22:59.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:22:59.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.96539 2023-12-02T01:23:01.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:01.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.96539 2023-12-02T01:23:03.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:03.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.96539 2023-12-02T01:23:04.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:04.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.96539 2023-12-02T01:23:05.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:05.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.96539 2023-12-02T01:23:07.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:07.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.96539 2023-12-02T01:23:09.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:09.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.96539 2023-12-02T01:23:10.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:10.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.96539 2023-12-02T01:23:12.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:12.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.96539 2023-12-02T01:23:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.96539 2023-12-02T01:23:15.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:15.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.96539 2023-12-02T01:23:16.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:16.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.96539 2023-12-02T01:23:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.96539 2023-12-02T01:23:19.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:19.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.96539 2023-12-02T01:23:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-12-02T01:23:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.96539 2023-12-02T01:23:22.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.96539 2023-12-02T01:23:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2023-12-02T01:23:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 4 2023-12-02T01:23:22.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2023-12-02T01:23:22.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2023-12-02T01:23:22.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-12-02T01:23:22.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-12-02T01:23:22.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-12-02T01:23:22.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-12-02T01:23:23.242 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:23:23.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2023-12-02T01:23:25.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2023-12-02T01:23:25.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2023-12-02T01:23:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2023-12-02T01:23:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-12-02T01:23:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-12-02T01:23:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-12-02T01:23:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-12-02T01:23:25.523 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:23:25.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2023-12-02T01:23:27.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2023-12-02T01:23:27.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2023-12-02T01:23:27.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2023-12-02T01:23:27.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-12-02T01:23:27.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-12-02T01:23:27.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-12-02T01:23:27.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-12-02T01:23:27.801 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:23:27.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2023-12-02T01:23:29.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2023-12-02T01:23:29.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2023-12-02T01:23:29.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2023-12-02T01:23:29.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-12-02T01:23:29.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-12-02T01:23:29.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-12-02T01:23:29.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-12-02T01:23:30.084 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:23:30.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2023-12-02T01:23:30.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2023-12-02T01:23:30.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2023-12-02T01:23:30.351 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:23:30.351 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2023-12-02T01:23:24.808685+0000 24'1300 28:1323 [1,0,2,3] 1 [1,0] 1 0'0 2023-12-02T01:20:44.976941+0000 0'0 2023-12-02T01:20:44.976941+0000 0 0 periodic scrub scheduled @ 2023-12-03T11:12:47.968709+0000 0 0 2023-12-02T01:23:30.351 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:23:30.351 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:23:30.352 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:23:30.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2023-12-02T01:23:40.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2023-12-02T01:23:40.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:23:40.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:23:40.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:23:40.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:23:40.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:23:45.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:23:45.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2023-12-02T01:23:45.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='not scheduling scrubs due to active recovery' 2023-12-02T01:23:45.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2023-12-02T01:23:45.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2023-12-02T01:23:45.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:26.545+0000 7fd8fea12640 15 osd.0 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:30.513+0000 7fd8fea12640 15 osd.0 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:35.616+0000 7fd8fea12640 15 osd.0 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:36.623+0000 7fd8fea12640 15 osd.0 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:38.564+0000 7fd8fea12640 15 osd.0 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:43.577+0000 7fd8fea12640 15 osd.0 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:30.893+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:34.935+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:35.887+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:38.899+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:39.850+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:40.863+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:41.886+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.700 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:42.851+0000 7fe1f0cec640 15 osd.1 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2023-12-02T01:23:45.703 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:27.563+0000 7f76b469c640 15 osd.2 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.703 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:29.546+0000 7f76b469c640 15 osd.2 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.703 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:30.517+0000 7f76b469c640 15 osd.2 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.703 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:31.531+0000 7f76b469c640 15 osd.2 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.703 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:33.525+0000 7f76b469c640 15 osd.2 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2023-12-02T01:23:45.705 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:27.110+0000 7f85f055e640 15 osd.3 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.705 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:29.099+0000 7f85f055e640 15 osd.3 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.706 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:31.175+0000 7f85f055e640 15 osd.3 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.706 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:38.236+0000 7f85f055e640 15 osd.3 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.706 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:39.255+0000 7f85f055e640 15 osd.3 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.706 INFO:tasks.workunit.client.0.smithi162.stdout:2023-12-02T01:23:41.300+0000 7f85f055e640 15 osd.3 osd-scrub:restrictions_on_scrubbing: not scheduling scrubs due to active recovery 2023-12-02T01:23:45.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2023-12-02T01:23:45.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2023-12-02T01:23:45.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2023-12-02T01:23:45.707 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2023-12-02T01:23:45.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2023-12-02T01:23:45.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2023-12-02T01:23:45.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-12-02T01:23:45.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2023-12-02T01:23:45.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2023-12-02T01:23:45.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2023-12-02T01:23:45.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-12-02T01:23:45.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2023-12-02T01:23:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2023-12-02T01:23:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2023-12-02T01:23:45.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-12-02T01:23:45.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2023-12-02T01:23:45.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2023-12-02T01:23:45.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:45.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2023-12-02T01:23:45.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-12-02T01:23:45.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2023-12-02T01:23:45.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:23:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:23:45.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:23:45.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:23:45.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:23:45.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:23:45.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:23:45.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:23:45.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:23:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:23:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:23:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:23:45.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:23:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:23:45.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-12-02T01:23:45.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-12-02T01:23:45.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:23:45.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:45.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:45.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96539 2023-12-02T01:23:45.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:23:45.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:23:45.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2023-12-02T01:23:45.766 INFO:tasks.workunit.client.0.smithi162.stdout:TEST PASSED 2023-12-02T01:23:45.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2023-12-02T01:23:45.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.96539 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-12-02T01:23:45.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:23:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:23:45.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:23:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:23:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:23:45.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:23:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:23:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:23:45.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:23:45.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:23:45.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:23:45.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-12-02T01:23:45.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-12-02T01:23:45.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:23:45.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:45.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:45.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96539 2023-12-02T01:23:45.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:23:45.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:23:45.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-12-02T01:23:45.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-12-02T01:23:45.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:45.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:45.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96539 2023-12-02T01:23:45.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-12-02T01:23:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-12-02T01:23:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-12-02T01:23:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-12-02T01:23:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-12-02T01:23:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-12-02T01:23:45.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-12-02T01:23:45.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-12-02T01:23:45.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-12-02T01:23:45.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-12-02T01:23:45.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:23:46.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-12-02T01:23:46.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-12-02T01:23:46.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-12-02T01:23:46.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-12-02T01:23:46.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-12-02T01:23:46.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-12-02T01:23:46.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-12-02T01:23:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-12-02T01:23:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-12-02T01:23:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-12-02T01:23:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:23:46.355 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:46.355 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:46.355 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96539/ceph-mon.a.asok 2023-12-02T01:23:46.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-12-02T01:23:46.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96539/ceph-mon.a.asok config get fsid 2023-12-02T01:23:46.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-12-02T01:23:46.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-12-02T01:23:46.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-12-02T01:23:46.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-12-02T01:23:46.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-12-02T01:23:46.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-12-02T01:23:46.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-12-02T01:23:46.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-12-02T01:23:46.431 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:23:46.431 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:46.431 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:46.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96539/ceph-mon.a.asok 2023-12-02T01:23:46.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-12-02T01:23:46.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96539/ceph-mon.a.asok config get mon_host 2023-12-02T01:23:46.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2023-12-02T01:23:46.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-12-02T01:23:46.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-12-02T01:23:46.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-12-02T01:23:46.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-12-02T01:23:46.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-12-02T01:23:46.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-12-02T01:23:47.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-12-02T01:23:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2023-12-02T01:23:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2023-12-02T01:23:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2023-12-02T01:23:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2023-12-02T01:23:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2023-12-02T01:23:47.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2023-12-02T01:23:47.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:47.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:23:47.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:23:47.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:23:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:23:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=919d146c-f01d-41b6-9144-57c97648de37 2023-12-02T01:23:47.355 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 919d146c-f01d-41b6-9144-57c97648de37 2023-12-02T01:23:47.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 919d146c-f01d-41b6-9144-57c97648de37' 2023-12-02T01:23:47.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:23:47.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjh2plP6ftFRAAuFUrt5JNOcyIovH0OcxHsw== 2023-12-02T01:23:47.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjh2plP6ftFRAAuFUrt5JNOcyIovH0OcxHsw=="}' 2023-12-02T01:23:47.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 919d146c-f01d-41b6-9144-57c97648de37 -i td/osd-recovery-scrub/0/new.json 2023-12-02T01:23:47.574 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:23:47.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-12-02T01:23:47.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAjh2plP6ftFRAAuFUrt5JNOcyIovH0OcxHsw== --osd-uuid 919d146c-f01d-41b6-9144-57c97648de37 2023-12-02T01:23:47.617 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:47.615+0000 7f7dffa14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:47.625 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:47.624+0000 7f7dffa14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:47.626 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:47.625+0000 7f7dffa14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:47.625+0000 7f7dffa14500 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-12-02T01:23:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:47.625+0000 7f7dffa14500 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-12-02T01:23:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:47.625+0000 7f7dffa14500 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-12-02T01:23:47.717 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:47.716+0000 7f7dffa14500 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-12-02T01:23:53.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-12-02T01:23:53.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:23:53.251 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2023-12-02T01:23:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-12-02T01:23:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:23:53.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-12-02T01:23:53.599 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2023-12-02T01:23:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:23:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:23:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:23:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:23:53.630 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:53.628+0000 7f63cb16c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:53.631 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:53.630+0000 7f63cb16c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:53.632 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:53.631+0000 7f63cb16c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:53.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-12-02T01:23:53.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:23:53.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-12-02T01:23:53.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:23:53.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:23:53.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:23:53.869 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:23:53.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:23:53.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:23:53.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:23:54.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:23:54.159 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:54.158+0000 7f63cb16c500 -1 Falling back to public interface 2023-12-02T01:23:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:23:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:23:55.138 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:23:55.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:23:55.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:23:55.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:23:55.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:23:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:23:56.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:23:56.415 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:23:56.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:23:56.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:23:56.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:23:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:56.539+0000 7f63cb16c500 -1 osd.0 0 log_to_monitors true 2023-12-02T01:23:56.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:23:57.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:23:57.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:23:57.733 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:23:57.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:23:57.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:23:57.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:23:58.082 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1637496609,v1:127.0.0.1:6803/1637496609] [v2:127.0.0.1:6804/1637496609,v1:127.0.0.1:6805/1637496609] exists,up 919d146c-f01d-41b6-9144-57c97648de37 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-12-02T01:23:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:23:58.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:23:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1fb58d3f-0cdb-428f-8190-76ea0aa85987 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1fb58d3f-0cdb-428f-8190-76ea0aa85987' 2023-12-02T01:23:58.090 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 1fb58d3f-0cdb-428f-8190-76ea0aa85987 2023-12-02T01:23:58.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:23:58.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuh2plFSsSBhAATDeMt2SYK1ivFLrKZ0YWnA== 2023-12-02T01:23:58.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuh2plFSsSBhAATDeMt2SYK1ivFLrKZ0YWnA=="}' 2023-12-02T01:23:58.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1fb58d3f-0cdb-428f-8190-76ea0aa85987 -i td/osd-recovery-scrub/1/new.json 2023-12-02T01:23:58.561 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:23:58.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-12-02T01:23:58.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAuh2plFSsSBhAATDeMt2SYK1ivFLrKZ0YWnA== --osd-uuid 1fb58d3f-0cdb-428f-8190-76ea0aa85987 2023-12-02T01:23:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:58.603+0000 7fb8b6c69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:58.645 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:58.644+0000 7fb8b6c69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:58.646 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:58.645+0000 7fb8b6c69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:23:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:58.645+0000 7fb8b6c69500 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-12-02T01:23:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:58.645+0000 7fb8b6c69500 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-12-02T01:23:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:58.645+0000 7fb8b6c69500 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-12-02T01:23:58.712 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:23:58.710+0000 7fb8b6c69500 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-12-02T01:24:04.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-12-02T01:24:04.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:24:04.799 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2023-12-02T01:24:04.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-12-02T01:24:04.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:24:05.145 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2023-12-02T01:24:05.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-12-02T01:24:05.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:05.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:24:05.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:24:05.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:24:05.174 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:05.173+0000 7f8a17cc7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:05.176 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:05.175+0000 7f8a17cc7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:05.177 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:05.176+0000 7f8a17cc7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:05.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-12-02T01:24:05.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:24:05.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-12-02T01:24:05.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:24:05.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:24:05.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:05.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:24:05.415 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:24:05.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:05.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:24:05.692 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:05.691+0000 7f8a17cc7500 -1 Falling back to public interface 2023-12-02T01:24:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:06.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:06.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:06.712 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:24:06.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:24:06.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:06.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:24:06.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:07.980 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:24:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:24:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:24:08.028 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:08.027+0000 7f8a17cc7500 -1 osd.1 0 log_to_monitors true 2023-12-02T01:24:08.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:09.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:09.255 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:24:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:24:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:24:09.542 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4126763533,v1:127.0.0.1:6811/4126763533] [v2:127.0.0.1:6812/4126763533,v1:127.0.0.1:6813/4126763533] exists,up 1fb58d3f-0cdb-428f-8190-76ea0aa85987 2023-12-02T01:24:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:24:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:24:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:24:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:24:09.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:24:09.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa3e45ed-be96-4344-88c0-eb179f1bd6e6 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 aa3e45ed-be96-4344-88c0-eb179f1bd6e6' 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 aa3e45ed-be96-4344-88c0-eb179f1bd6e6 2023-12-02T01:24:09.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:24:09.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5h2plMWCAIRAAtkoaPGMN8C7ing5rzCAFxg== 2023-12-02T01:24:09.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5h2plMWCAIRAAtkoaPGMN8C7ing5rzCAFxg=="}' 2023-12-02T01:24:09.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa3e45ed-be96-4344-88c0-eb179f1bd6e6 -i td/osd-recovery-scrub/2/new.json 2023-12-02T01:24:09.923 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:24:09.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-12-02T01:24:09.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQA5h2plMWCAIRAAtkoaPGMN8C7ing5rzCAFxg== --osd-uuid aa3e45ed-be96-4344-88c0-eb179f1bd6e6 2023-12-02T01:24:09.966 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:09.964+0000 7fb8bec0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:10.008 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:10.007+0000 7fb8bec0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:10.009 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:10.008+0000 7fb8bec0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:10.009 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:10.008+0000 7fb8bec0c500 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-12-02T01:24:10.009 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:10.008+0000 7fb8bec0c500 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-12-02T01:24:10.009 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:10.008+0000 7fb8bec0c500 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-12-02T01:24:10.083 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:10.081+0000 7fb8bec0c500 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-12-02T01:24:16.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-12-02T01:24:16.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:24:16.183 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2023-12-02T01:24:16.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-12-02T01:24:16.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:24:16.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-12-02T01:24:16.528 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2023-12-02T01:24:16.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:16.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:24:16.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:24:16.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:24:16.558 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:16.556+0000 7faecddbc500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:16.559 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:16.558+0000 7faecddbc500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:16.559+0000 7faecddbc500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:16.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-12-02T01:24:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:24:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-12-02T01:24:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:24:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:24:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:16.797 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:24:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:24:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:24:17.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:17.081 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:17.080+0000 7faecddbc500 -1 Falling back to public interface 2023-12-02T01:24:18.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:18.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:18.065 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:24:18.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:24:18.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:18.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:24:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:19.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:19.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:19.333 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:24:19.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:24:19.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:19.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:24:19.483 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:19.481+0000 7faecddbc500 -1 osd.2 0 log_to_monitors true 2023-12-02T01:24:19.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:20.642 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:24:20.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:24:20.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:20.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:24:20.911 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3228100358,v1:127.0.0.1:6819/3228100358] [v2:127.0.0.1:6820/3228100358,v1:127.0.0.1:6821/3228100358] exists,up aa3e45ed-be96-4344-88c0-eb179f1bd6e6 2023-12-02T01:24:20.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:24:20.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-12-02T01:24:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:24:20.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:24:20.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:24:20.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:24:20.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:24:20.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:24:20.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:24:20.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-12-02T01:24:20.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:24:20.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5e62d2e-119c-4a04-bedd-9ec38ce0746f 2023-12-02T01:24:20.919 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 d5e62d2e-119c-4a04-bedd-9ec38ce0746f 2023-12-02T01:24:20.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d5e62d2e-119c-4a04-bedd-9ec38ce0746f' 2023-12-02T01:24:20.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:24:20.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEh2plCAp7NxAAVxw3M8HDsikrO25iAeNdww== 2023-12-02T01:24:20.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEh2plCAp7NxAAVxw3M8HDsikrO25iAeNdww=="}' 2023-12-02T01:24:20.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5e62d2e-119c-4a04-bedd-9ec38ce0746f -i td/osd-recovery-scrub/3/new.json 2023-12-02T01:24:21.288 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:24:21.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-12-02T01:24:21.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBEh2plCAp7NxAAVxw3M8HDsikrO25iAeNdww== --osd-uuid d5e62d2e-119c-4a04-bedd-9ec38ce0746f 2023-12-02T01:24:21.332 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:21.331+0000 7f1c36e14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:21.373 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:21.372+0000 7f1c36e14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:21.374 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:21.373+0000 7f1c36e14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:21.374 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:21.373+0000 7f1c36e14500 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-12-02T01:24:21.375 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:21.373+0000 7f1c36e14500 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-12-02T01:24:21.375 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:21.373+0000 7f1c36e14500 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-12-02T01:24:21.431 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:21.430+0000 7f1c36e14500 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-12-02T01:24:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-12-02T01:24:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:24:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-12-02T01:24:27.407 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2023-12-02T01:24:27.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:24:27.753 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2023-12-02T01:24:27.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-12-02T01:24:27.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:27.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:24:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:24:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:24:27.782 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:27.781+0000 7fd269e14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:27.784 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:27.783+0000 7fd269e14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:27.785 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:27.784+0000 7fd269e14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:28.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-12-02T01:24:28.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:24:28.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-12-02T01:24:28.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:24:28.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:24:28.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:28.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:24:28.022 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:24:28.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:28.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:24:28.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:28.304 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:28.303+0000 7fd269e14500 -1 Falling back to public interface 2023-12-02T01:24:29.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:29.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:29.298 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:24:29.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:24:29.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:29.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:24:29.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:30.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:30.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:30.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:24:30.579 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:24:30.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:30.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:24:30.634 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:30.632+0000 7fd269e14500 -1 osd.3 0 log_to_monitors true 2023-12-02T01:24:30.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:31.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:31.859 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:24:31.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:31.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:24:31.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:31.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:24:32.152 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/423945687,v1:127.0.0.1:6827/423945687] [v2:127.0.0.1:6828/423945687,v1:127.0.0.1:6829/423945687] exists,up d5e62d2e-119c-4a04-bedd-9ec38ce0746f 2023-12-02T01:24:32.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:24:32.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2023-12-02T01:24:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:24:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c13a1711-132c-4c99-83e6-ec2d40cd641b 2023-12-02T01:24:32.158 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 c13a1711-132c-4c99-83e6-ec2d40cd641b 2023-12-02T01:24:32.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c13a1711-132c-4c99-83e6-ec2d40cd641b' 2023-12-02T01:24:32.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:24:32.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQh2ploz9AChAAqCdEZn6EQxhtbmpoeJ/u7w== 2023-12-02T01:24:32.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQh2ploz9AChAAqCdEZn6EQxhtbmpoeJ/u7w=="}' 2023-12-02T01:24:32.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c13a1711-132c-4c99-83e6-ec2d40cd641b -i td/osd-recovery-scrub/4/new.json 2023-12-02T01:24:32.537 INFO:tasks.workunit.client.0.smithi162.stdout:4 2023-12-02T01:24:32.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2023-12-02T01:24:32.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBQh2ploz9AChAAqCdEZn6EQxhtbmpoeJ/u7w== --osd-uuid c13a1711-132c-4c99-83e6-ec2d40cd641b 2023-12-02T01:24:32.580 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:32.579+0000 7f2c4fe0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:32.622 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:32.620+0000 7f2c4fe0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:32.622 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:32.621+0000 7f2c4fe0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:32.623 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:32.622+0000 7f2c4fe0e500 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-12-02T01:24:32.623 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:32.622+0000 7f2c4fe0e500 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-12-02T01:24:32.623 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:32.622+0000 7f2c4fe0e500 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-12-02T01:24:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:32.712+0000 7f2c4fe0e500 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2023-12-02T01:24:38.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2023-12-02T01:24:38.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:24:38.740 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2023-12-02T01:24:38.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2023-12-02T01:24:38.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:24:39.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2023-12-02T01:24:39.086 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2023-12-02T01:24:39.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:39.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:24:39.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:24:39.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:24:39.116 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:39.114+0000 7f8c4c225500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:39.117 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:39.116+0000 7f8c4c225500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:39.119 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:39.117+0000 7f8c4c225500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:39.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2023-12-02T01:24:39.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:24:39.641 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:39.640+0000 7f8c4c225500 -1 Falling back to public interface 2023-12-02T01:24:39.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:40.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:40.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:40.653 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:24:40.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:24:40.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:40.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:24:40.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:41.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:41.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:41.930 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:24:41.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:24:41.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:41.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:24:42.016 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:42.015+0000 7f8c4c225500 -1 osd.4 0 log_to_monitors true 2023-12-02T01:24:42.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:43.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:43.226 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:24:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:24:43.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:43.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:24:43.560 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3651869816,v1:127.0.0.1:6835/3651869816] [v2:127.0.0.1:6836/3651869816,v1:127.0.0.1:6837/3651869816] exists,up c13a1711-132c-4c99-83e6-ec2d40cd641b 2023-12-02T01:24:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:24:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:24:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:24:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2023-12-02T01:24:43.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:24:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:24:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:24:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:24:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:24:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:24:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:24:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:24:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:24:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:24:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:24:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2023-12-02T01:24:43.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:24:43.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=15b16ddc-3675-4777-bd0b-6a03a06f4aed 2023-12-02T01:24:43.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 15b16ddc-3675-4777-bd0b-6a03a06f4aed' 2023-12-02T01:24:43.573 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 15b16ddc-3675-4777-bd0b-6a03a06f4aed 2023-12-02T01:24:43.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:24:43.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbh2plOSWLIhAA6yO54x26m3+TWZjYONOcjg== 2023-12-02T01:24:43.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbh2plOSWLIhAA6yO54x26m3+TWZjYONOcjg=="}' 2023-12-02T01:24:43.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 15b16ddc-3675-4777-bd0b-6a03a06f4aed -i td/osd-recovery-scrub/5/new.json 2023-12-02T01:24:43.907 INFO:tasks.workunit.client.0.smithi162.stdout:5 2023-12-02T01:24:43.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2023-12-02T01:24:43.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBbh2plOSWLIhAA6yO54x26m3+TWZjYONOcjg== --osd-uuid 15b16ddc-3675-4777-bd0b-6a03a06f4aed 2023-12-02T01:24:43.951 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:43.950+0000 7f8661816500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:43.992 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:43.990+0000 7f8661816500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:43.991+0000 7f8661816500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:43.992+0000 7f8661816500 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-12-02T01:24:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:43.992+0000 7f8661816500 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-12-02T01:24:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:43.992+0000 7f8661816500 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-12-02T01:24:44.092 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:44.091+0000 7f8661816500 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2023-12-02T01:24:50.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2023-12-02T01:24:50.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:24:50.294 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2023-12-02T01:24:50.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2023-12-02T01:24:50.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:24:50.639 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2023-12-02T01:24:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2023-12-02T01:24:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:24:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:24:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:24:50.670 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:50.668+0000 7ff93c320500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:50.671 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:50.670+0000 7ff93c320500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:50.672 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:50.671+0000 7ff93c320500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2023-12-02T01:24:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:24:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2023-12-02T01:24:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:24:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:24:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:50.911 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:24:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:24:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:24:51.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:51.191 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:51.190+0000 7ff93c320500 -1 Falling back to public interface 2023-12-02T01:24:52.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:52.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:52.178 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:24:52.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:24:52.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:52.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:24:52.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:53.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:53.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:53.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:24:53.455 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:24:53.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:53.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:24:53.631 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:53.630+0000 7ff93c320500 -1 osd.5 0 log_to_monitors true 2023-12-02T01:24:53.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:24:54.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:24:54.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:24:54.839 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:24:54.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:24:54.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:24:54.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:24:55.130 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1737018443,v1:127.0.0.1:6843/1737018443] [v2:127.0.0.1:6844/1737018443,v1:127.0.0.1:6845/1737018443] exists,up 15b16ddc-3675-4777-bd0b-6a03a06f4aed 2023-12-02T01:24:55.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:24:55.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:24:55.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:24:55.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:24:55.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:24:55.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:24:55.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:24:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:24:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:24:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:24:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2023-12-02T01:24:55.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:24:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44a6254b-173a-46b8-96c8-989cd9172018 2023-12-02T01:24:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 44a6254b-173a-46b8-96c8-989cd9172018' 2023-12-02T01:24:55.137 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 44a6254b-173a-46b8-96c8-989cd9172018 2023-12-02T01:24:55.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:24:55.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnh2plycv6CBAA53haRDzIgjAhPDv+Mq8ARQ== 2023-12-02T01:24:55.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnh2plycv6CBAA53haRDzIgjAhPDv+Mq8ARQ=="}' 2023-12-02T01:24:55.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44a6254b-173a-46b8-96c8-989cd9172018 -i td/osd-recovery-scrub/6/new.json 2023-12-02T01:24:55.468 INFO:tasks.workunit.client.0.smithi162.stdout:6 2023-12-02T01:24:55.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2023-12-02T01:24:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBnh2plycv6CBAA53haRDzIgjAhPDv+Mq8ARQ== --osd-uuid 44a6254b-173a-46b8-96c8-989cd9172018 2023-12-02T01:24:55.512 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:55.510+0000 7f10ae3ad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:55.519 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:55.518+0000 7f10ae3ad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:55.520 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:55.519+0000 7f10ae3ad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:24:55.521 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:55.520+0000 7f10ae3ad500 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-12-02T01:24:55.521 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:55.520+0000 7f10ae3ad500 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-12-02T01:24:55.521 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:55.520+0000 7f10ae3ad500 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-12-02T01:24:55.645 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:24:55.644+0000 7f10ae3ad500 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2023-12-02T01:25:01.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2023-12-02T01:25:01.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:25:01.397 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2023-12-02T01:25:01.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2023-12-02T01:25:01.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:25:01.769 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2023-12-02T01:25:01.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2023-12-02T01:25:01.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:25:01.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:25:01.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:25:01.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:25:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:01.798+0000 7fe5e819b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:01.801 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:01.800+0000 7fe5e819b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:01.802 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:01.801+0000 7fe5e819b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:02.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:25:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:02.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-12-02T01:25:02.321 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:02.320+0000 7fe5e819b500 -1 Falling back to public interface 2023-12-02T01:25:02.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:25:03.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:25:03.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:03.325 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:25:03.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:25:03.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:03.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-12-02T01:25:03.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:25:04.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:25:04.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:04.630 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:25:04.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:25:04.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:04.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-12-02T01:25:04.642 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:04.640+0000 7fe5e819b500 -1 osd.6 0 log_to_monitors true 2023-12-02T01:25:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:25:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:25:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:05.977 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:25:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:25:05.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:05.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-12-02T01:25:06.245 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1688877172,v1:127.0.0.1:6851/1688877172] [v2:127.0.0.1:6852/1688877172,v1:127.0.0.1:6853/1688877172] exists,up 44a6254b-173a-46b8-96c8-989cd9172018 2023-12-02T01:25:06.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:25:06.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2023-12-02T01:25:06.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:25:06.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-12-02T01:25:06.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-12-02T01:25:06.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:25:06.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:25:06.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:25:06.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:25:06.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:25:06.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-12-02T01:25:06.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2023-12-02T01:25:06.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:25:06.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bad25b56-abb4-4034-bb0f-4cdeed0ac33e 2023-12-02T01:25:06.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 bad25b56-abb4-4034-bb0f-4cdeed0ac33e' 2023-12-02T01:25:06.259 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 bad25b56-abb4-4034-bb0f-4cdeed0ac33e 2023-12-02T01:25:06.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:25:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByh2pltn3WDxAA2tslGRJTlXtQB1tkXB3kcg== 2023-12-02T01:25:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByh2pltn3WDxAA2tslGRJTlXtQB1tkXB3kcg=="}' 2023-12-02T01:25:06.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bad25b56-abb4-4034-bb0f-4cdeed0ac33e -i td/osd-recovery-scrub/7/new.json 2023-12-02T01:25:06.647 INFO:tasks.workunit.client.0.smithi162.stdout:7 2023-12-02T01:25:06.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2023-12-02T01:25:06.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQByh2pltn3WDxAA2tslGRJTlXtQB1tkXB3kcg== --osd-uuid bad25b56-abb4-4034-bb0f-4cdeed0ac33e 2023-12-02T01:25:06.691 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:06.690+0000 7f4bdf4c9500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:06.724 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:06.723+0000 7f4bdf4c9500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:06.725 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:06.724+0000 7f4bdf4c9500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:06.725 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:06.724+0000 7f4bdf4c9500 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-12-02T01:25:06.725 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:06.724+0000 7f4bdf4c9500 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-12-02T01:25:06.725 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:06.724+0000 7f4bdf4c9500 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-12-02T01:25:06.807 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:06.806+0000 7f4bdf4c9500 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2023-12-02T01:25:13.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2023-12-02T01:25:13.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:25:13.128 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2023-12-02T01:25:13.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2023-12-02T01:25:13.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:25:13.505 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2023-12-02T01:25:13.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2023-12-02T01:25:13.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=20982e6b-0205-4565-a21a-74ffe1370591 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.96539/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-12-02T01:25:13.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:25:13.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:25:13.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:25:13.535 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:13.534+0000 7fde8f49e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:13.537 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:13.536+0000 7fde8f49e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:13.538 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:13.537+0000 7fde8f49e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:25:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2023-12-02T01:25:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:25:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2023-12-02T01:25:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:25:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:25:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:25:13.780 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:25:13.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:13.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-12-02T01:25:14.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:25:14.073 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:14.072+0000 7fde8f49e500 -1 Falling back to public interface 2023-12-02T01:25:15.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:25:15.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:15.046 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:25:15.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:25:15.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:15.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-12-02T01:25:15.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:25:16.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:25:16.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:16.315 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:25:16.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:25:16.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:16.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-12-02T01:25:16.403 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:25:16.402+0000 7fde8f49e500 -1 osd.7 0 log_to_monitors true 2023-12-02T01:25:16.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:25:17.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:25:17.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:25:17.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:25:17.596 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:25:17.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:25:17.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-12-02T01:25:17.876 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3114708651,v1:127.0.0.1:6859/3114708651] [v2:127.0.0.1:6860/3114708651,v1:127.0.0.1:6861/3114708651] exists,up bad25b56-abb4-4034-bb0f-4cdeed0ac33e 2023-12-02T01:25:17.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:25:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:25:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:25:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2023-12-02T01:25:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2023-12-02T01:25:18.350 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test' created 2023-12-02T01:25:18.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-12-02T01:25:19.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2023-12-02T01:25:19.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-12-02T01:25:19.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-12-02T01:25:19.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-12-02T01:25:19.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-12-02T01:25:19.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-12-02T01:25:19.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-12-02T01:25:19.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-12-02T01:25:19.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-12-02T01:25:19.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-12-02T01:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-12-02T01:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-12-02T01:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-12-02T01:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-12-02T01:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2023-12-02T01:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2023-12-02T01:25:19.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:1 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:2 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:3 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:4 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:5 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:6 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:19.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-12-02T01:25:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2023-12-02T01:25:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2023-12-02T01:25:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2023-12-02T01:25:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:20.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-12-02T01:25:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2023-12-02T01:25:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2023-12-02T01:25:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2023-12-02T01:25:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:20.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-12-02T01:25:20.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2023-12-02T01:25:20.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2023-12-02T01:25:20.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453' 2023-12-02T01:25:20.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:20.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-12-02T01:25:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2023-12-02T01:25:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2023-12-02T01:25:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931' 2023-12-02T01:25:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:20.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-12-02T01:25:21.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2023-12-02T01:25:21.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2023-12-02T01:25:21.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409' 2023-12-02T01:25:21.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:21.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2023-12-02T01:25:21.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2023-12-02T01:25:21.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2023-12-02T01:25:21.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887' 2023-12-02T01:25:21.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:21.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2023-12-02T01:25:21.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2023-12-02T01:25:21.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2023-12-02T01:25:21.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365' 2023-12-02T01:25:21.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:25:21.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2023-12-02T01:25:21.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2023-12-02T01:25:21.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2023-12-02T01:25:21.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365 7-171798691842' 2023-12-02T01:25:21.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:21.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2023-12-02T01:25:21.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:21.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2023-12-02T01:25:21.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2023-12-02T01:25:21.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:21.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2023-12-02T01:25:21.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2023-12-02T01:25:21.443 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836498 2023-12-02T01:25:21.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2023-12-02T01:25:21.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2023-12-02T01:25:21.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:21.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2023-12-02T01:25:21.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:21.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2023-12-02T01:25:21.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2023-12-02T01:25:21.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:21.716 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672976 2023-12-02T01:25:21.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2023-12-02T01:25:21.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2023-12-02T01:25:21.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2023-12-02T01:25:21.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2023-12-02T01:25:21.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:21.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2023-12-02T01:25:21.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:21.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2023-12-02T01:25:21.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2023-12-02T01:25:21.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:21.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2023-12-02T01:25:21.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2023-12-02T01:25:21.991 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509453 2023-12-02T01:25:21.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2023-12-02T01:25:22.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2023-12-02T01:25:22.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:22.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2023-12-02T01:25:22.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2023-12-02T01:25:22.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2023-12-02T01:25:22.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:22.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2023-12-02T01:25:22.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2023-12-02T01:25:22.264 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345931 2023-12-02T01:25:22.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2023-12-02T01:25:22.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2023-12-02T01:25:22.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:22.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2023-12-02T01:25:22.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:22.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2023-12-02T01:25:22.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2023-12-02T01:25:22.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:22.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2023-12-02T01:25:22.540 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182409 2023-12-02T01:25:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2023-12-02T01:25:22.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2023-12-02T01:25:22.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2023-12-02T01:25:22.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:22.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2023-12-02T01:25:22.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:22.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2023-12-02T01:25:22.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2023-12-02T01:25:22.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:22.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2023-12-02T01:25:22.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2023-12-02T01:25:22.812 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018887 2023-12-02T01:25:22.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2023-12-02T01:25:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2023-12-02T01:25:23.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:23.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855365 2023-12-02T01:25:23.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:23.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2023-12-02T01:25:23.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855365 2023-12-02T01:25:23.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:23.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2023-12-02T01:25:23.084 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855365 2023-12-02T01:25:23.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2023-12-02T01:25:23.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2023-12-02T01:25:23.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2023-12-02T01:25:23.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:25:23.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2023-12-02T01:25:23.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:25:23.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2023-12-02T01:25:23.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2023-12-02T01:25:23.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:25:23.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2023-12-02T01:25:23.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2023-12-02T01:25:23.357 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691842 2023-12-02T01:25:23.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2023-12-02T01:25:23.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2023-12-02T01:25:23.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-12-02T01:25:23.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-12-02T01:25:23.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-12-02T01:25:24.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-12-02T01:25:24.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2023-12-02T01:25:24.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-12-02T01:25:24.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-12-02T01:25:24.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-12-02T01:25:24.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2023-12-02T01:25:24.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-12-02T01:25:24.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-12-02T01:25:24.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2023-12-02T01:25:24.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2023-12-02T01:25:24.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2023-12-02T01:25:24.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2023-12-02T01:25:24.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.96539 bs=1M count=50 2023-12-02T01:25:25.008 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records in 2023-12-02T01:25:25.008 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records out 2023-12-02T01:25:25.008 INFO:tasks.workunit.client.0.smithi162.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.143204 s, 366 MB/s 2023-12-02T01:25:25.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2023-12-02T01:25:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.96539 2023-12-02T01:25:26.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:26.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.96539 2023-12-02T01:25:28.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:28.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.96539 2023-12-02T01:25:29.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:29.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.96539 2023-12-02T01:25:31.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:31.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.96539 2023-12-02T01:25:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.96539 2023-12-02T01:25:34.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:34.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.96539 2023-12-02T01:25:35.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:35.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.96539 2023-12-02T01:25:37.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:37.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.96539 2023-12-02T01:25:38.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:38.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.96539 2023-12-02T01:25:40.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:40.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.96539 2023-12-02T01:25:41.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:41.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.96539 2023-12-02T01:25:43.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:43.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.96539 2023-12-02T01:25:44.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:44.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.96539 2023-12-02T01:25:46.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:46.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.96539 2023-12-02T01:25:47.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:47.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.96539 2023-12-02T01:25:49.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:49.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.96539 2023-12-02T01:25:50.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:50.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.96539 2023-12-02T01:25:52.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:52.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.96539 2023-12-02T01:25:54.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:54.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.96539 2023-12-02T01:25:55.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:55.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.96539 2023-12-02T01:25:57.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:57.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.96539 2023-12-02T01:25:58.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:25:58.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.96539 2023-12-02T01:26:00.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:00.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.96539 2023-12-02T01:26:02.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:02.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.96539 2023-12-02T01:26:03.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:03.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.96539 2023-12-02T01:26:04.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:04.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.96539 2023-12-02T01:26:06.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:06.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.96539 2023-12-02T01:26:08.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:08.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.96539 2023-12-02T01:26:09.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:09.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.96539 2023-12-02T01:26:11.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:11.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.96539 2023-12-02T01:26:12.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:12.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.96539 2023-12-02T01:26:14.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:14.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.96539 2023-12-02T01:26:15.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:15.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.96539 2023-12-02T01:26:17.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:17.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.96539 2023-12-02T01:26:18.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:18.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.96539 2023-12-02T01:26:20.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:20.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.96539 2023-12-02T01:26:21.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:21.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.96539 2023-12-02T01:26:23.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:23.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.96539 2023-12-02T01:26:24.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-12-02T01:26:24.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.96539 2023-12-02T01:26:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.96539 2023-12-02T01:26:26.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2023-12-02T01:26:26.719 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 3 2023-12-02T01:26:26.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2023-12-02T01:25:20.239621+0000 47'26 47:53 [5] 5 [5] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.364721+0000 47'13 47:38 [3] 3 [3] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:19.818530+0000 47'13 47:27 [1] 1 [1] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:19.675933+0000 47'13 47:30 [1] 1 [1] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:19.676044+0000 47'13 47:30 [1] 1 [1] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:25:19.676232+0000 0'0 47:17 [1] 1 [1] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2023-12-02T01:25:20.238823+0000 47'26 47:44 [5] 5 [5] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.364581+0000 47'13 47:32 [6] 6 [6] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.6 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.364672+0000 47'13 47:32 [6] 6 [6] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.989 INFO:tasks.workunit.client.0.smithi162.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2023-12-02T01:25:20.238762+0000 47'39 47:57 [4] 4 [4] 4 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2023-12-02T01:25:20.199853+0000 47'52 47:71 [3] 3 [3] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2023-12-02T01:25:20.239197+0000 47'52 47:70 [4] 4 [4] 4 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.238772+0000 47'13 47:33 [0] 0 [0] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:25:20.364419+0000 0'0 47:17 [3] 3 [3] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.239446+0000 47'13 47:40 [5] 5 [5] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.399442+0000 47'13 47:27 [0] 0 [0] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:19.636201+0000 47'13 47:36 [7] 7 [7] 7 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:19.793428+0000 47'13 47:27 [7] 7 [7] 7 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:19.676334+0000 47'13 47:29 [1] 1 [1] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.199893+0000 47'13 47:29 [6] 6 [6] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.239552+0000 47'13 47:30 [5] 5 [5] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.399337+0000 47'13 47:29 [0] 0 [0] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.990 INFO:tasks.workunit.client.0.smithi162.stdout:1.12 2 0 0 0 0 94371840 0 0 23 0 23 active+clean 2023-12-02T01:25:20.239380+0000 47'23 47:38 [0] 0 [0] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.238892+0000 47'13 47:27 [4] 4 [4] 4 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:25:20.238740+0000 0'0 47:14 [2] 2 [2] 2 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2023-12-02T01:25:19.818530+0000 47'39 47:54 [1] 1 [1] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:25:20.364681+0000 0'0 47:15 [6] 6 [6] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.399318+0000 47'13 47:28 [0] 0 [0] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:25:20.238914+0000 0'0 47:16 [5] 5 [5] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.364404+0000 47'13 47:28 [3] 3 [3] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:25:19.818725+0000 0'0 47:15 [1] 1 [1] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-12-02T01:25:20.399427+0000 47'13 47:28 [0] 0 [0] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:26:26.991 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:26:27.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2023-12-02T01:26:27.166 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:26:27.167 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_recovery_max_active": "0" 2023-12-02T01:26:27.167 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:26:27.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2023-12-02T01:26:27.474 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:26:27.474 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_recovery_max_active_hdd": "3" 2023-12-02T01:26:27.474 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:26:27.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2023-12-02T01:26:27.768 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:26:27.768 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_recovery_max_active_ssd": "10" 2023-12-02T01:26:27.768 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:26:27.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2023-12-02T01:26:27.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-12-02T01:26:27.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-12-02T01:26:27.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-12-02T01:26:27.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-12-02T01:26:27.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-12-02T01:26:28.119 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:26:28.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2023-12-02T01:26:28.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-12-02T01:26:30.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2023-12-02T01:26:30.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2023-12-02T01:26:30.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2023-12-02T01:26:30.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-12-02T01:26:30.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-12-02T01:26:30.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-12-02T01:26:30.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-12-02T01:26:30.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-12-02T01:26:30.398 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:26:30.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2023-12-02T01:26:30.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-12-02T01:26:32.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2023-12-02T01:26:32.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2023-12-02T01:26:32.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2023-12-02T01:26:32.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-12-02T01:26:32.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-12-02T01:26:32.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-12-02T01:26:32.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-12-02T01:26:32.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-12-02T01:26:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:26:32.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-12-02T01:26:32.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-12-02T01:26:34.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2023-12-02T01:26:34.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2023-12-02T01:26:34.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2023-12-02T01:26:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-12-02T01:26:34.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-12-02T01:26:34.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-12-02T01:26:34.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-12-02T01:26:34.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-12-02T01:26:34.958 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:26:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 3 -ge 2 2023-12-02T01:26:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2023-12-02T01:26:34.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2023-12-02T01:26:28.634114+0000 47'26 50:63 [5,0,2] 5 [5,0,2] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.826726+0000 47'13 50:48 [3,0,7] 3 [3,0,7] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2023-12-02T01:26:28.507847+0000 47'26 50:49 [1,3,0] 1 [1,3,0] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.876896+0000 47'13 50:38 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.633160+0000 47'13 50:38 [1,6,2] 1 [1,6,2] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:26:28.552334+0000 0'0 50:27 [1,4,5] 1 [1,4,5] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovering+degraded 2023-12-02T01:26:28.553204+0000 47'26 50:54 [5,4,3] 5 [5,4,3] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:29.003009+0000 47'13 50:42 [6,7,3] 6 [6,7,3] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.827210+0000 47'13 50:42 [6,7,3] 6 [6,7,3] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.221 INFO:tasks.workunit.client.0.smithi162.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 activating 2023-12-02T01:26:28.212760+0000 47'39 50:61 [4,2,6] 4 [4,2,6] 4 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2023-12-02T01:26:28.675027+0000 47'52 50:80 [3,0,6] 3 [3,0,6] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 activating+degraded 2023-12-02T01:26:28.551221+0000 47'52 50:74 [4,1,2] 4 [4,1,2] 4 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:29.207464+0000 47'13 50:44 [0,7,1] 0 [0,7,1] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:26:28.372430+0000 0'0 50:27 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.675037+0000 47'13 50:50 [5,0,4] 5 [5,0,4] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.728612+0000 47'13 50:37 [0,3,4] 0 [0,3,4] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2023-12-02T01:26:28.508564+0000 47'13 50:48 [7,0,2] 7 [7,0,2] 7 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.552581+0000 47'13 50:36 [7,4,2] 7 [7,4,2] 7 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.633714+0000 47'13 50:37 [1,4,2] 1 [1,4,2] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.508718+0000 47'13 50:38 [6,0,3] 6 [6,0,3] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:29.003241+0000 47'13 50:39 [5,7,6] 5 [5,7,6] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.675311+0000 47'13 50:38 [0,1,3] 0 [0,1,3] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovering+degraded 2023-12-02T01:26:28.508581+0000 47'26 50:52 [0,1,6] 0 [0,1,6] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.222 INFO:tasks.workunit.client.0.smithi162.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 activating 2023-12-02T01:26:28.212875+0000 47'13 50:31 [4,6,3] 4 [4,6,3] 4 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:26:28.876916+0000 0'0 50:25 [2,0,4] 2 [2,0,4] 2 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2023-12-02T01:26:28.675038+0000 47'39 50:62 [1,5,3] 1 [1,5,3] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:26:28.827692+0000 0'0 50:26 [6,0,7] 6 [6,0,7] 6 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.918842+0000 47'13 50:37 [0,4,5] 0 [0,4,5] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:26:28.466303+0000 0'0 50:27 [5,2,3] 5 [5,2,3] 5 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.876744+0000 47'13 50:36 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-12-02T01:26:28.827218+0000 0'0 50:25 [1,7,4] 1 [1,7,4] 1 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2023-12-02T01:26:28.633137+0000 47'13 50:37 [0,1,2] 0 [0,1,2] 0 0'0 2023-12-02T01:25:18.234362+0000 0'0 2023-12-02T01:25:18.234362+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:25:18.234362+0000 0 0 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:26:35.223 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:26:35.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2023-12-02T01:26:35.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2023-12-02T01:26:35.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2023-12-02T01:26:35.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2023-12-02T01:26:35.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2023-12-02T01:26:35.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2023-12-02T01:26:35.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 117996"' 2023-12-02T01:26:35.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 117996' 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/117999: /' 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118001"' 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118001' 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118006: /' 2023-12-02T01:26:35.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2023-12-02T01:26:35.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2023-12-02T01:26:35.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118008"' 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118008' 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118013: /' 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118015"' 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118015' 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2023-12-02T01:26:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2023-12-02T01:26:35.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2023-12-02T01:26:35.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118022"' 2023-12-02T01:26:35.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118022' 2023-12-02T01:26:35.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118026"' 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118026' 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118020: /' 2023-12-02T01:26:35.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2023-12-02T01:26:35.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118029: /' 2023-12-02T01:26:35.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2023-12-02T01:26:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118034: /' 2023-12-02T01:26:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2023-12-02T01:26:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118035"' 2023-12-02T01:26:35.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118035' 2023-12-02T01:26:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2023-12-02T01:26:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118039: /' 2023-12-02T01:26:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2023-12-02T01:26:35.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2023-12-02T01:26:35.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2023-12-02T01:26:35.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118045"' 2023-12-02T01:26:35.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118045' 2023-12-02T01:26:35.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118048: /' 2023-12-02T01:26:35.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2023-12-02T01:26:35.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2023-12-02T01:26:35.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2023-12-02T01:26:35.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118052"' 2023-12-02T01:26:35.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118052' 2023-12-02T01:26:35.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2023-12-02T01:26:35.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2023-12-02T01:26:35.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118054"' 2023-12-02T01:26:35.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118054' 2023-12-02T01:26:35.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2023-12-02T01:26:35.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2023-12-02T01:26:35.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118060"' 2023-12-02T01:26:35.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118060' 2023-12-02T01:26:35.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118057: /' 2023-12-02T01:26:35.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2023-12-02T01:26:35.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2023-12-02T01:26:35.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118059: /' 2023-12-02T01:26:35.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2023-12-02T01:26:35.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2023-12-02T01:26:35.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118066"' 2023-12-02T01:26:35.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118066' 2023-12-02T01:26:35.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118065: /' 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118075"' 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118075' 2023-12-02T01:26:35.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118073: /' 2023-12-02T01:26:35.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2023-12-02T01:26:35.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2023-12-02T01:26:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2023-12-02T01:26:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118080"' 2023-12-02T01:26:35.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118080' 2023-12-02T01:26:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2023-12-02T01:26:35.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2023-12-02T01:26:35.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118088"' 2023-12-02T01:26:35.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118085: /' 2023-12-02T01:26:35.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118088' 2023-12-02T01:26:35.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2023-12-02T01:26:35.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118087: /' 2023-12-02T01:26:35.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2023-12-02T01:26:35.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2023-12-02T01:26:35.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2023-12-02T01:26:35.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118096"' 2023-12-02T01:26:35.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118096' 2023-12-02T01:26:35.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2023-12-02T01:26:35.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2023-12-02T01:26:35.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118100"' 2023-12-02T01:26:35.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118100' 2023-12-02T01:26:35.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118104: /' 2023-12-02T01:26:35.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2023-12-02T01:26:35.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2023-12-02T01:26:35.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2023-12-02T01:26:35.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118114: /' 2023-12-02T01:26:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2023-12-02T01:26:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118115"' 2023-12-02T01:26:35.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118115' 2023-12-02T01:26:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118106: /' 2023-12-02T01:26:35.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2023-12-02T01:26:35.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2023-12-02T01:26:35.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118133"' 2023-12-02T01:26:35.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118133' 2023-12-02T01:26:35.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118126: /' 2023-12-02T01:26:35.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2023-12-02T01:26:35.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2023-12-02T01:26:35.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2023-12-02T01:26:35.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118139: /' 2023-12-02T01:26:35.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2023-12-02T01:26:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118146"' 2023-12-02T01:26:35.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118146' 2023-12-02T01:26:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2023-12-02T01:26:35.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2023-12-02T01:26:35.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2023-12-02T01:26:35.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118177"' 2023-12-02T01:26:35.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118177' 2023-12-02T01:26:35.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2023-12-02T01:26:35.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118176: /' 2023-12-02T01:26:35.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2023-12-02T01:26:35.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2023-12-02T01:26:35.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118195"' 2023-12-02T01:26:35.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118195' 2023-12-02T01:26:35.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118193: /' 2023-12-02T01:26:35.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2023-12-02T01:26:35.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2023-12-02T01:26:35.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2023-12-02T01:26:35.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118213"' 2023-12-02T01:26:35.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118213' 2023-12-02T01:26:35.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2023-12-02T01:26:35.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2023-12-02T01:26:35.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2023-12-02T01:26:35.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118248"' 2023-12-02T01:26:35.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118248' 2023-12-02T01:26:35.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2023-12-02T01:26:35.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118247: /' 2023-12-02T01:26:35.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2023-12-02T01:26:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2023-12-02T01:26:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118220: /' 2023-12-02T01:26:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118278"' 2023-12-02T01:26:35.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118278' 2023-12-02T01:26:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2023-12-02T01:26:35.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2023-12-02T01:26:35.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118288"' 2023-12-02T01:26:35.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118288' 2023-12-02T01:26:35.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2023-12-02T01:26:35.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118280: /' 2023-12-02T01:26:35.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118291: /' 2023-12-02T01:26:35.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2023-12-02T01:26:35.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2023-12-02T01:26:35.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2023-12-02T01:26:35.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118298"' 2023-12-02T01:26:35.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118298' 2023-12-02T01:26:35.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2023-12-02T01:26:35.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2023-12-02T01:26:35.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118305"' 2023-12-02T01:26:35.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118305' 2023-12-02T01:26:35.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118301: /' 2023-12-02T01:26:35.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2023-12-02T01:26:35.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118308: /' 2023-12-02T01:26:35.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2023-12-02T01:26:35.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2023-12-02T01:26:35.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2023-12-02T01:26:35.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118330"' 2023-12-02T01:26:35.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118330' 2023-12-02T01:26:35.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2023-12-02T01:26:35.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118342: /' 2023-12-02T01:26:35.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2023-12-02T01:26:35.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2023-12-02T01:26:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2023-12-02T01:26:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118355"' 2023-12-02T01:26:35.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118355' 2023-12-02T01:26:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118340: /' 2023-12-02T01:26:35.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2023-12-02T01:26:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2023-12-02T01:26:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118360"' 2023-12-02T01:26:35.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118360' 2023-12-02T01:26:35.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2023-12-02T01:26:35.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2023-12-02T01:26:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2023-12-02T01:26:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2023-12-02T01:26:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2023-12-02T01:26:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 118363"' 2023-12-02T01:26:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 118363' 2023-12-02T01:26:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2023-12-02T01:26:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 117996 118001 118008 118015 118022 118026 118035 118045 118052 118054 118060 118066 118075 118080 118088 118096 118100 118115 118133 118146 118177 118195 118213 118248 118278 118288 118298 118305 118330 118355 118360 118363' 2023-12-02T01:26:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2023-12-02T01:26:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 117996 2023-12-02T01:26:35.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2023-12-02T01:26:35.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118375: /' 2023-12-02T01:26:35.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118371: /' 2023-12-02T01:26:35.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2023-12-02T01:26:35.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2023-12-02T01:26:35.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/118369: /' 2023-12-02T01:26:41.722 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: instructing pg 1.0 on osd.7 to scrub 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: dumped pgs 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:41.725 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-12-02T01:26:41.725 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-12-02T01:26:41.725 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:41.725 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.380 INFO:tasks.workunit.client.0.smithi162.stderr:117999: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_sta118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2023-12-02T01:26:42.380 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2023-12-02T01:26:42.380 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2023-12-02T01:26:42.380 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: instructing pg 1.12 on osd.0 to scrub 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: dumped pgs 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_la118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: instructing pg 1.8 on osd.5 to scrub 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: dumped pgs 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2023-12-02T01:26:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2023-12-02T01:26:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2023-12-02T01:26:42.727 INFO:tasks.workunit.client.0.smithi162.stderr:118057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2023-12-02T01:26:42.727 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2023-12-02T01:26:42.727 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2023-12-02T01:26:42.727 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.727 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: instructing pg 1.3 on osd.4 to scrub 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: dumped pgs 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:118020: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:42.980 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: instructing pg 1.11 on osd.0 to scrub 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: dumped pgs 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:42.981 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:118126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: instructing pg 1.6 on osd.6 to scrub 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: dumped pgs 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:42.989 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:118039: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:43.600 INFO:tasks.workunit.client.0.smithi162.stderr:mp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-12-02T01:26:43.600 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:43.600 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:43.600 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2023-12-02T01:26:43.600 INFO:tasks.workunit.client.0.smithi162.stderr:117999: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2023-12-02T01:26:43.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2023-12-02T01:26:43.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2023-12-02T01:26:43.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2023-12-02T01:26:43.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2023-12-02T01:26:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2023-12-02T01:26:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2023-12-02T01:26:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2023-12-02T01:26:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2023-12-02T01:26:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: instructing pg 1.4 on osd.3 to scrub 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: dumped pgs 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:118034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:43.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2023-12-02T01:26:43.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:43.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118001 2023-12-02T01:26:43.667 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2023-12-02T01:26:43.667 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2023-12-02T01:26:43.667 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2023-12-02T01:26:43.667 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.667 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2023-12-02T01:26:43.667 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.667 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: instructing pg 1.7 on osd.6 to scrub 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: dumped pgs 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:118048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:43.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:43.782 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: instructing pg 1.16 on osd.6 to scrub 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: dumped pgs 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2023-12-02T01:26:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2023-12-02T01:26:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:118247: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2023-12-02T01:26:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:26:40.242558+0000 '>' '' 2023-12-02T01:26:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:118247: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:43.939 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2023-12-02T01:26:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:118065: instructing pg 1.a on osd.1 to scrub 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: dumped pgs 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2023-12-02T01:26:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.942 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2023-12-02T01:26:43.942 INFO:tasks.workunit.client.0.smithi162.stderr:118065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.942 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:43.942 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:43.942 INFO:tasks.workunit.client.0.smithi162.stderr:118065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:43.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2023-12-02T01:26:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2023-12-02T01:26:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2023-12-02T01:26:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: instructing pg 1.2 on osd.0 to scrub 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: dumped pgs 2023-12-02T01:26:43.979 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:43.981 INFO:tasks.workunit.client.0.smithi162.stderr:118013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:43.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2023-12-02T01:26:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: instructing pg 1.f on osd.6 to scrub 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: dumped pgs 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.052 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:118114: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2023-12-02T01:26:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2023-12-02T01:26:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: instructing pg 1.1c on osd.0 to scrub 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.271 INFO:tasks.workunit.client.0.smithi162.stderr:118342: dumped pgs 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:118342: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2023-12-02T01:26:44.325 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: instructing pg 1.1b on osd.0 to scrub 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: dumped pgs 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.326 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:118340: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.364 INFO:tasks.workunit.client.0.smithi162.stderr:-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.364 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.364 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.364 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2023-12-02T01:26:44.364 INFO:tasks.workunit.client.0.smithi162.stderr:118057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2023-12-02T01:26:44.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2023-12-02T01:26:44.438 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2023-12-02T01:26:44.438 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: instructing pg 1.b on osd.1 to scrub 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.439 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: dumped pgs 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2023-12-02T01:26:44.440 INFO:tasks.workunit.client.0.smithi162.stderr:118073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.441 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: instructing pg 1.1f on osd.5 to scrub 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: dumped pgs 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2023-12-02T01:26:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:118369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: instructing pg 1.18 on osd.5 to scrub 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: dumped pgs 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.452 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:26:39.265635+0000 '>' '' 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:118291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.468 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2023-12-02T01:26:44.468 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: instructing pg 1.1a on osd.1 to scrub 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: dumped pgs 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2023-12-02T01:26:44.470 INFO:tasks.workunit.client.0.smithi162.stderr:118308: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:118308: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: instructing pg 1.5 on osd.4 to scrub 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: dumped pgs 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2023-12-02T01:26:44.497 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.499 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.499 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.499 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.499 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.499 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.499 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2023-12-02T01:26:44.499 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2023-12-02T01:26:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:118029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2023-12-02T01:26:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:118029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.520 INFO:tasks.workunit.client.0.smithi162.stderr:st_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2023-12-02T01:26:44.520 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.520 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.520 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2023-12-02T01:26:44.520 INFO:tasks.workunit.client.0.smithi162.stderr:118139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2023-12-02T01:26:44.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2023-12-02T01:26:44.538 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.539 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: instructing pg 1.e on osd.1 to scrub 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: dumped pgs 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.540 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:118104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:118104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.553 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2023-12-02T01:26:44.553 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: instructing pg 1.9 on osd.1 to scrub 2023-12-02T01:26:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: dumped pgs 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2023-12-02T01:26:44.555 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.556 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2023-12-02T01:26:44.556 INFO:tasks.workunit.client.0.smithi162.stderr:118059: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.556 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.556 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.556 INFO:tasks.workunit.client.0.smithi162.stderr:118059: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.566 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2023-12-02T01:26:44.566 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2023-12-02T01:26:44.566 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2023-12-02T01:26:44.566 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: instructing pg 1.c on osd.3 to scrub 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: dumped pgs 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:26:40.606292+0000 '>' '' 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.568 INFO:tasks.workunit.client.0.smithi162.stderr:118085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.593 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2023-12-02T01:26:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:118193: instructing pg 1.14 on osd.2 to scrub 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: dumped pgs 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2023-12-02T01:26:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2023-12-02T01:26:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2023-12-02T01:26:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:118193: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:26:40.682617+0000 '>' '' 2023-12-02T01:26:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:118193: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.618 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2023-12-02T01:26:44.618 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2023-12-02T01:26:44.618 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2023-12-02T01:26:44.618 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: instructing pg 1.19 on osd.3 to scrub 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.619 INFO:tasks.workunit.client.0.smithi162.stderr:118301: dumped pgs 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.620 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.621 INFO:tasks.workunit.client.0.smithi162.stderr:118301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.651 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2023-12-02T01:26:44.651 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2023-12-02T01:26:44.651 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2023-12-02T01:26:44.651 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.651 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.651 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: instructing pg 1.d on osd.7 to scrub 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: dumped pgs 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.652 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.654 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: instructing pg 1.1d on osd.1 to scrub 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: dumped pgs 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.655 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone./home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:118375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: instructing pg 1.15 on osd.1 to scrub 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: dumped pgs 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.659 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:118220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.665 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2023-12-02T01:26:44.666 INFO:tasks.workunit.client.0.smithi162.stderr:118006: instructing pg 1.1 on osd.5 to scrub 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: dumped pgs 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.667 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.668 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.669 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.669 INFO:tasks.workunit.client.0.smithi162.stderr:118006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118008 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118015 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118022 2023-12-02T01:26:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118026 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118035 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118045 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118052 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118054 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118060 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118066 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118075 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118080 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118088 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118096 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118100 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.687 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2023-12-02T01:26:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:118280: instructing pg 1.17 on osd.0 to scrub 2023-12-02T01:26:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.688 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: dumped pgs 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.689 INFO:tasks.workunit.client.0.smithi162.stderr:118280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2023-12-02T01:26:44.690 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.690 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.690 INFO:tasks.workunit.client.0.smithi162.stderr:118280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.711 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: instructing pg 1.13 on osd.4 to scrub 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: dumped pgs 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2023-12-02T01:26:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2023-12-02T01:26:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2023-12-02T01:26:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:118176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:118176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.736 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2023-12-02T01:26:44.736 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2023-12-02T01:26:44.736 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2023-12-02T01:26:44.736 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.736 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.736 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: instructing pg 1.10 on osd.5 to scrub 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: dumped pgs 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2023-12-02T01:26:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:118106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118115 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118133 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2023-12-02T01:26:44.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 2 + 1 2023-12-02T01:26:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=3 2023-12-02T01:26:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118146 2023-12-02T01:26:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118177 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118195 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118213 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118248 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118278 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118288 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118298 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118305 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118330 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118355 2023-12-02T01:26:44.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118360 2023-12-02T01:26:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2023-12-02T01:26:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2023-12-02T01:26:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2023-12-02T01:26:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: instructing pg 1.1e on osd.3 to scrub 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: dumped pgs 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2023-12-02T01:26:44.751 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2023-12-02T01:25:18.234362+0000 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2023-12-02T01:25:18.234362+0000 '>' '' 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:118371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 118363 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 3 -eq 0 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2023-12-02T01:26:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/mon.a.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.5.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.6.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.1.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/mgr.x.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.0.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.4.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.3.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.7.pid 2023-12-02T01:26:44.754 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.2.pid' 2023-12-02T01:26:44.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.2.pid 2023-12-02T01:26:44.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='105141 2023-12-02T01:26:44.755 INFO:tasks.workunit.client.0.smithi162.stderr:113052 2023-12-02T01:26:44.755 INFO:tasks.workunit.client.0.smithi162.stderr:114428 2023-12-02T01:26:44.755 INFO:tasks.workunit.client.0.smithi162.stderr:107548 2023-12-02T01:26:44.755 INFO:tasks.workunit.client.0.smithi162.stderr:105225 2023-12-02T01:26:44.755 INFO:tasks.workunit.client.0.smithi162.stderr:106171 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:111676 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:110300 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:115804 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:108924' 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 105141 113052 114428 107548 105225 106171 111676 110300 115804 108924 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:26:44.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:26:50.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:26:50.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2023-12-02T01:26:50.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2023-12-02T01:26:50.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2023-12-02T01:26:50.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2023-12-02T01:26:50.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2023-12-02T01:26:50.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2023-12-02T01:26:50.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2023-12-02T01:26:50.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2023-12-02T01:26:50.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2023-12-02T01:26:50.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2023-12-02T01:26:50.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2023-12-02T01:26:50.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2023-12-02T01:26:50.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2023-12-02T01:26:50.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2023-12-02T01:26:50.103 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2023-12-02T01:26:50.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2023-12-02T01:26:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2023-12-02T01:26:50.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2023-12-02T01:26:50.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2023-12-02T01:26:50.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2023-12-02T01:26:50.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2023-12-02T01:26:50.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2023-12-02T01:26:50.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2023-12-02T01:26:50.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:50.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2023-12-02T01:26:50.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2023-12-02T01:26:50.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2023-12-02T01:26:50.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-12-02T01:26:50.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-12-02T01:26:50.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-12-02T01:26:50.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:26:50.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:26:50.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:26:50.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:26:50.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:26:50.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:26:50.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:26:50.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:26:50.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:26:50.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:26:50.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:26:50.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:26:50.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:26:50.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:26:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:26:50.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:26:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:26:50.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-12-02T01:26:50.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-12-02T01:26:50.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:26:50.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:50.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:26:50.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96539 2023-12-02T01:26:50.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:26:50.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:26:50.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2023-12-02T01:26:50.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2023-12-02T01:26:50.173 INFO:tasks.workunit.client.0.smithi162.stdout:TEST PASSED 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:26:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:26:50.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:26:50.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:26:50.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:26:50.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:26:50.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:26:50.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:26:50.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:26:50.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:26:50.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:26:50.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:26:50.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:26:50.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:26:50.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-12-02T01:26:50.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-12-02T01:26:50.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:26:50.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:50.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96539 2023-12-02T01:26:50.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96539 2023-12-02T01:26:50.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:26:50.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:26:50.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2023-12-02T01:26:50.183 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-02T01:26:50.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-02T01:26:50.252 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2023-12-02T01:26:50.252 DEBUG:teuthology.orchestra.run.smithi162:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2023-12-02T01:26:50.314 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-12-02T01:26:50.319 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:25: run: local dir=td/osd-scrub-dump 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:26: run: shift 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:27: run: local CHUNK_MAX=5 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:29: run: export CEPH_MON=127.0.0.1:7184 2023-12-02T01:26:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:29: run: CEPH_MON=127.0.0.1:7184 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: export CEPH_ARGS 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: uuidgen 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: CEPH_ARGS+='--fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none ' 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7184 ' 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:33: run: CEPH_ARGS+='--osd_max_scrubs=4 ' 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:34: run: CEPH_ARGS+='--osd_shallow_scrub_chunk_max=5 ' 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:35: run: CEPH_ARGS+='--osd_scrub_sleep=3 ' 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:36: run: CEPH_ARGS+='--osd_pool_default_size=3 ' 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:40: run: CEPH_ARGS+='--osd_op_queue=wpq ' 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:42: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: set 2023-12-02T01:26:50.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: local funcs=TEST_recover_unexpected 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:44: run: for func in $funcs 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:45: run: setup td/osd-scrub-dump 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-dump 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-dump 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:26:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:26:50.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:26:50.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:26:50.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:26:50.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:26:50.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:26:50.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:26:50.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:26:50.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:26:50.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:26:50.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:26:50.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:26:50.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:26:50.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:26:50.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-12-02T01:26:50.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2023-12-02T01:26:50.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:26:50.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:50.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:26:50.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.121566 2023-12-02T01:26:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:26:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:26:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-dump 2023-12-02T01:26:50.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-12-02T01:26:50.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:50.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:26:50.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.121566 2023-12-02T01:26:50.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-12-02T01:26:50.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-dump 1' TERM HUP INT 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:46: run: TEST_recover_unexpected td/osd-scrub-dump 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:52: TEST_recover_unexpected: local dir=td/osd-scrub-dump 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:53: TEST_recover_unexpected: shift 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:54: TEST_recover_unexpected: local OSDS=6 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:55: TEST_recover_unexpected: local PGS=16 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:56: TEST_recover_unexpected: local POOLS=3 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:57: TEST_recover_unexpected: local OBJS=1000 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: run_mon td/osd-scrub-dump a 2023-12-02T01:26:50.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-dump 2023-12-02T01:26:50.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-12-02T01:26:50.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-12-02T01:26:50.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-12-02T01:26:50.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-dump/a 2023-12-02T01:26:50.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-dump/a --run-dir=td/osd-scrub-dump 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:26:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-dump/a '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-dump/log --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-12-02T01:26:50.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-12-02T01:26:50.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-12-02T01:26:50.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-mon.a.asok 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-12-02T01:26:50.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.121566/ceph-mon.a.asok config get fsid 2023-12-02T01:26:50.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-12-02T01:26:50.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-12-02T01:26:50.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-12-02T01:26:50.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-mon.a.asok 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-12-02T01:26:50.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.121566/ceph-mon.a.asok config get mon_host 2023-12-02T01:26:51.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:60: TEST_recover_unexpected: run_mgr td/osd-scrub-dump x 2023-12-02T01:26:51.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-dump 2023-12-02T01:26:51.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-12-02T01:26:51.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-12-02T01:26:51.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-12-02T01:26:51.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-dump/x 2023-12-02T01:26:51.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-12-02T01:26:51.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-12-02T01:26:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-dump/x '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-12-02T01:26:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:26:51.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: seq 0 5 2023-12-02T01:26:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:26:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 0 2023-12-02T01:26:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/0 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/0' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/0/journal' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:26:51.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:26:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:26:51.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-12-02T01:26:51.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/0 2023-12-02T01:26:51.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:26:51.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=befbb1a9-a7f7-48e9-aac9-bb5f63c73af1 2023-12-02T01:26:51.910 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 befbb1a9-a7f7-48e9-aac9-bb5f63c73af1 2023-12-02T01:26:51.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 befbb1a9-a7f7-48e9-aac9-bb5f63c73af1' 2023-12-02T01:26:51.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:26:51.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbh2plm/duNxAAGTTceD8TP8JZ8LtRff0nXA== 2023-12-02T01:26:51.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbh2plm/duNxAAGTTceD8TP8JZ8LtRff0nXA=="}' 2023-12-02T01:26:51.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new befbb1a9-a7f7-48e9-aac9-bb5f63c73af1 -i td/osd-scrub-dump/0/new.json 2023-12-02T01:26:52.183 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:26:52.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/0/new.json 2023-12-02T01:26:52.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbh2plm/duNxAAGTTceD8TP8JZ8LtRff0nXA== --osd-uuid befbb1a9-a7f7-48e9-aac9-bb5f63c73af1 2023-12-02T01:26:52.230 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:52.229+0000 7f9e3c265500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:26:52.233 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:52.232+0000 7f9e3c265500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:26:52.234 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:52.233+0000 7f9e3c265500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:26:52.235 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:52.234+0000 7f9e3c265500 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2023-12-02T01:26:52.235 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:52.234+0000 7f9e3c265500 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2023-12-02T01:26:52.235 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:52.234+0000 7f9e3c265500 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2023-12-02T01:26:52.342 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:52.341+0000 7f9e3c265500 -1 bluestore(td/osd-scrub-dump/0) _read_fsid unparsable uuid 2023-12-02T01:26:57.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/0/keyring 2023-12-02T01:26:57.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:26:57.688 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2023-12-02T01:26:57.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-12-02T01:26:57.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:26:58.064 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2023-12-02T01:26:58.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-12-02T01:26:58.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:26:58.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:26:58.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:26:58.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:26:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:58.096+0000 7fdef13e0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:26:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:58.098+0000 7fdef13e0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:26:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:58.099+0000 7fdef13e0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:26:58.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-12-02T01:26:58.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:26:58.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-12-02T01:26:58.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:26:58.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:26:58.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:26:58.342 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:26:58.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:26:58.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:26:58.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:26:58.618 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:26:58.614+0000 7fdef13e0500 -1 Falling back to public interface 2023-12-02T01:26:58.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:26:59.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:26:59.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:26:59.655 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:26:59.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:26:59.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:26:59.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:26:59.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:00.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:00.936 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:27:00.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:00.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:27:00.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:00.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:27:00.977 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:00.975+0000 7fdef13e0500 -1 osd.0 0 log_to_monitors true 2023-12-02T01:27:01.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:02.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:02.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:02.278 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:27:02.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:27:02.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:02.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-12-02T01:27:02.561 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3397845935,v1:127.0.0.1:6803/3397845935] [v2:127.0.0.1:6804/3397845935,v1:127.0.0.1:6805/3397845935] exists,up befbb1a9-a7f7-48e9-aac9-bb5f63c73af1 2023-12-02T01:27:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:27:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:27:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:27:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:27:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 1 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/1 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/1' 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/1/journal' 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:27:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:27:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-12-02T01:27:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/1 2023-12-02T01:27:02.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:27:02.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4db21964-d3b2-41cf-8fd8-b7934251bea6 2023-12-02T01:27:02.568 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 4db21964-d3b2-41cf-8fd8-b7934251bea6 2023-12-02T01:27:02.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4db21964-d3b2-41cf-8fd8-b7934251bea6' 2023-12-02T01:27:02.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:27:02.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmh2plVnr0IhAA7PhwTnvrJnXRqYUxO+KRNg== 2023-12-02T01:27:02.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmh2plVnr0IhAA7PhwTnvrJnXRqYUxO+KRNg=="}' 2023-12-02T01:27:02.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4db21964-d3b2-41cf-8fd8-b7934251bea6 -i td/osd-scrub-dump/1/new.json 2023-12-02T01:27:02.913 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:27:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/1/new.json 2023-12-02T01:27:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmh2plVnr0IhAA7PhwTnvrJnXRqYUxO+KRNg== --osd-uuid 4db21964-d3b2-41cf-8fd8-b7934251bea6 2023-12-02T01:27:02.960 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:02.958+0000 7f25a1814500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:02.997 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:02.996+0000 7f25a1814500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:02.998 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:02.997+0000 7f25a1814500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:02.999 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:02.997+0000 7f25a1814500 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2023-12-02T01:27:02.999 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:02.997+0000 7f25a1814500 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2023-12-02T01:27:02.999 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:02.997+0000 7f25a1814500 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2023-12-02T01:27:03.063 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:03.063+0000 7f25a1814500 -1 bluestore(td/osd-scrub-dump/1) _read_fsid unparsable uuid 2023-12-02T01:27:09.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/1/keyring 2023-12-02T01:27:09.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:27:09.001 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2023-12-02T01:27:09.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-12-02T01:27:09.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:27:09.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-12-02T01:27:09.361 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2023-12-02T01:27:09.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:27:09.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:27:09.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:27:09.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:27:09.395 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:09.393+0000 7f4dbe715500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:09.396 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:09.395+0000 7f4dbe715500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:09.397 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:09.396+0000 7f4dbe715500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:09.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-12-02T01:27:09.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:27:09.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-12-02T01:27:09.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:27:09.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:27:09.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:09.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:27:09.640 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:27:09.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:09.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:27:09.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:09.917+0000 7f4dbe715500 -1 Falling back to public interface 2023-12-02T01:27:10.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:10.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:10.913 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:27:10.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:27:10.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:10.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:27:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:12.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:12.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:12.221 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:27:12.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:27:12.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:12.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:27:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:12.373+0000 7f4dbe715500 -1 osd.1 0 log_to_monitors true 2023-12-02T01:27:12.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:13.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:13.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:13.584 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:27:13.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:27:13.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:13.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-12-02T01:27:13.926 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/366326595,v1:127.0.0.1:6811/366326595] [v2:127.0.0.1:6812/366326595,v1:127.0.0.1:6813/366326595] exists,up 4db21964-d3b2-41cf-8fd8-b7934251bea6 2023-12-02T01:27:13.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 2 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/2 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:27:13.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/2' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/2/journal' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:27:13.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:27:13.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:27:13.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:27:13.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-12-02T01:27:13.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/2 2023-12-02T01:27:13.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:27:13.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7329fd0-5b1e-455f-a95e-16967e800337 2023-12-02T01:27:13.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f7329fd0-5b1e-455f-a95e-16967e800337' 2023-12-02T01:27:13.935 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 f7329fd0-5b1e-455f-a95e-16967e800337 2023-12-02T01:27:13.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:27:13.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxh2pltrLBOBAAUYxPVBuQfbEjjAc3OK4jRA== 2023-12-02T01:27:13.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxh2pltrLBOBAAUYxPVBuQfbEjjAc3OK4jRA=="}' 2023-12-02T01:27:13.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7329fd0-5b1e-455f-a95e-16967e800337 -i td/osd-scrub-dump/2/new.json 2023-12-02T01:27:14.278 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:27:14.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/2/new.json 2023-12-02T01:27:14.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxh2pltrLBOBAAUYxPVBuQfbEjjAc3OK4jRA== --osd-uuid f7329fd0-5b1e-455f-a95e-16967e800337 2023-12-02T01:27:14.325 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:14.324+0000 7f45815e2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:14.362 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:14.362+0000 7f45815e2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:14.363 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:14.363+0000 7f45815e2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:14.363 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:14.363+0000 7f45815e2500 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2023-12-02T01:27:14.363 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:14.363+0000 7f45815e2500 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2023-12-02T01:27:14.363 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:14.363+0000 7f45815e2500 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2023-12-02T01:27:14.429 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:14.428+0000 7f45815e2500 -1 bluestore(td/osd-scrub-dump/2) _read_fsid unparsable uuid 2023-12-02T01:27:20.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/2/keyring 2023-12-02T01:27:20.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:27:20.419 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2023-12-02T01:27:20.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-12-02T01:27:20.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:27:20.778 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2023-12-02T01:27:20.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-12-02T01:27:20.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:27:20.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:27:20.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:27:20.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:27:20.812 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:20.811+0000 7efcd5249500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:20.813 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:20.813+0000 7efcd5249500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:20.814 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:20.814+0000 7efcd5249500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:21.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-12-02T01:27:21.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:27:21.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-12-02T01:27:21.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:27:21.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:27:21.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:21.099 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:27:21.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:27:21.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:21.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:27:21.328 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:21.328+0000 7efcd5249500 -1 Falling back to public interface 2023-12-02T01:27:21.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:22.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:22.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:22.412 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:27:22.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:27:22.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:22.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:27:22.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:23.700 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:27:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:27:23.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:23.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:27:23.743 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:23.743+0000 7efcd5249500 -1 osd.2 0 log_to_monitors true 2023-12-02T01:27:24.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:25.069 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:27:25.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:27:25.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:25.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-12-02T01:27:25.352 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2873202789,v1:127.0.0.1:6819/2873202789] [v2:127.0.0.1:6820/2873202789,v1:127.0.0.1:6821/2873202789] exists,up f7329fd0-5b1e-455f-a95e-16967e800337 2023-12-02T01:27:25.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:27:25.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:27:25.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:27:25.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:27:25.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 3 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/3 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/3' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/3/journal' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2023-12-02T01:27:25.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:27:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/3 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fdd52330-a795-488b-abd1-9a03250c5e66 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 fdd52330-a795-488b-abd1-9a03250c5e66 2023-12-02T01:27:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 fdd52330-a795-488b-abd1-9a03250c5e66' 2023-12-02T01:27:25.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:27:25.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9h2plxCiSFhAAZa36YcI1wEMNbvxrP/xzjQ== 2023-12-02T01:27:25.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9h2plxCiSFhAAZa36YcI1wEMNbvxrP/xzjQ=="}' 2023-12-02T01:27:25.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fdd52330-a795-488b-abd1-9a03250c5e66 -i td/osd-scrub-dump/3/new.json 2023-12-02T01:27:25.728 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:27:25.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/3/new.json 2023-12-02T01:27:25.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9h2plxCiSFhAAZa36YcI1wEMNbvxrP/xzjQ== --osd-uuid fdd52330-a795-488b-abd1-9a03250c5e66 2023-12-02T01:27:25.776 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:25.775+0000 7f2fb02d2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:25.822 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:25.821+0000 7f2fb02d2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:25.822 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:25.822+0000 7f2fb02d2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:25.823 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:25.823+0000 7f2fb02d2500 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2023-12-02T01:27:25.823 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:25.823+0000 7f2fb02d2500 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2023-12-02T01:27:25.823 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:25.823+0000 7f2fb02d2500 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2023-12-02T01:27:25.913 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:25.913+0000 7f2fb02d2500 -1 bluestore(td/osd-scrub-dump/3) _read_fsid unparsable uuid 2023-12-02T01:27:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/3/keyring 2023-12-02T01:27:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:27:31.700 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2023-12-02T01:27:31.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-12-02T01:27:31.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:27:32.051 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2023-12-02T01:27:32.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-12-02T01:27:32.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:27:32.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:27:32.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:27:32.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:27:32.085 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:32.083+0000 7fcf47818500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:32.095 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:32.094+0000 7fcf47818500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:32.096 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:32.095+0000 7fcf47818500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:27:32.329 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:27:32.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:32.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:27:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:32.628 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:32.627+0000 7fcf47818500 -1 Falling back to public interface 2023-12-02T01:27:33.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:33.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:33.607 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:27:33.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:27:33.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:33.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:27:33.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:34.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:34.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:34.888 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:27:34.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:27:34.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:34.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:27:34.952 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:34.950+0000 7fcf47818500 -1 osd.3 0 log_to_monitors true 2023-12-02T01:27:35.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:36.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:36.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:36.164 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:27:36.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:27:36.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:36.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-12-02T01:27:36.462 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2765422977,v1:127.0.0.1:6827/2765422977] [v2:127.0.0.1:6828/2765422977,v1:127.0.0.1:6829/2765422977] exists,up fdd52330-a795-488b-abd1-9a03250c5e66 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 4 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/4 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:27:36.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/4' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/4/journal' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:27:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/4 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:27:36.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1e6e28e-a518-4c61-9969-0983eef533c0 2023-12-02T01:27:36.469 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 e1e6e28e-a518-4c61-9969-0983eef533c0 2023-12-02T01:27:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 e1e6e28e-a518-4c61-9969-0983eef533c0' 2023-12-02T01:27:36.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:27:36.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIiGplzh4eHRAANoe7dZXjd0wbc9qoFHTAPA== 2023-12-02T01:27:36.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIiGplzh4eHRAANoe7dZXjd0wbc9qoFHTAPA=="}' 2023-12-02T01:27:36.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1e6e28e-a518-4c61-9969-0983eef533c0 -i td/osd-scrub-dump/4/new.json 2023-12-02T01:27:36.828 INFO:tasks.workunit.client.0.smithi162.stdout:4 2023-12-02T01:27:36.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/4/new.json 2023-12-02T01:27:36.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIiGplzh4eHRAANoe7dZXjd0wbc9qoFHTAPA== --osd-uuid e1e6e28e-a518-4c61-9969-0983eef533c0 2023-12-02T01:27:36.875 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:36.874+0000 7f99b6a16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:36.912+0000 7f99b6a16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:36.914 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:36.913+0000 7f99b6a16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:36.914 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:36.913+0000 7f99b6a16500 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2023-12-02T01:27:36.915 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:36.913+0000 7f99b6a16500 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2023-12-02T01:27:36.915 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:36.913+0000 7f99b6a16500 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2023-12-02T01:27:37.029 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:37.028+0000 7f99b6a16500 -1 bluestore(td/osd-scrub-dump/4) _read_fsid unparsable uuid 2023-12-02T01:27:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/4/keyring 2023-12-02T01:27:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:27:43.118 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2023-12-02T01:27:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2023-12-02T01:27:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:27:43.470 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2023-12-02T01:27:43.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2023-12-02T01:27:43.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:27:43.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:27:43.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:27:43.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:27:43.505 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:43.504+0000 7f1d89c1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:43.507 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:43.506+0000 7f1d89c1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:43.508 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:43.507+0000 7f1d89c1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2023-12-02T01:27:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:27:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2023-12-02T01:27:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:27:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:27:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:27:43.750 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:27:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:27:44.028 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:44.027+0000 7f1d89c1a500 -1 Falling back to public interface 2023-12-02T01:27:44.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:45.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:45.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:45.097 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:27:45.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:27:45.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:45.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:27:45.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:46.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:46.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:46.379 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:27:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:27:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:27:46.488 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:46.486+0000 7f1d89c1a500 -1 osd.4 0 log_to_monitors true 2023-12-02T01:27:46.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:47.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:47.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:47.697 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:27:47.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:27:47.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:47.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-12-02T01:27:48.162 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1689601674,v1:127.0.0.1:6835/1689601674] [v2:127.0.0.1:6836/1689601674,v1:127.0.0.1:6837/1689601674] exists,up e1e6e28e-a518-4c61-9969-0983eef533c0 2023-12-02T01:27:48.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:27:48.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:27:48.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:27:48.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:27:48.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 5 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/5 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/5' 2023-12-02T01:27:48.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/5/journal' 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' 2023-12-02T01:27:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/5 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-12-02T01:27:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a01f5b57-1736-475b-bd61-7206afedb70b 2023-12-02T01:27:48.170 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 a01f5b57-1736-475b-bd61-7206afedb70b 2023-12-02T01:27:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a01f5b57-1736-475b-bd61-7206afedb70b' 2023-12-02T01:27:48.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-12-02T01:27:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUiGplgL01CxAAxoyZh1UENPJzpfJvGc6X2w== 2023-12-02T01:27:48.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUiGplgL01CxAAxoyZh1UENPJzpfJvGc6X2w=="}' 2023-12-02T01:27:48.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a01f5b57-1736-475b-bd61-7206afedb70b -i td/osd-scrub-dump/5/new.json 2023-12-02T01:27:48.526 INFO:tasks.workunit.client.0.smithi162.stdout:5 2023-12-02T01:27:48.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/5/new.json 2023-12-02T01:27:48.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUiGplgL01CxAAxoyZh1UENPJzpfJvGc6X2w== --osd-uuid a01f5b57-1736-475b-bd61-7206afedb70b 2023-12-02T01:27:48.573 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:48.571+0000 7f3c1c4c3500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:48.600+0000 7f3c1c4c3500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:48.601+0000 7f3c1c4c3500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:48.602+0000 7f3c1c4c3500 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2023-12-02T01:27:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:48.602+0000 7f3c1c4c3500 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2023-12-02T01:27:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:48.602+0000 7f3c1c4c3500 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2023-12-02T01:27:48.668 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:48.667+0000 7f3c1c4c3500 -1 bluestore(td/osd-scrub-dump/5) _read_fsid unparsable uuid 2023-12-02T01:27:54.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/5/keyring 2023-12-02T01:27:54.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-12-02T01:27:54.698 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2023-12-02T01:27:54.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2023-12-02T01:27:54.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-12-02T01:27:55.055 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2023-12-02T01:27:55.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2023-12-02T01:27:55.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32a30e14-41eb-4e0f-8eb0-c4036d94642d --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_shallow_scrub_chunk_max=5 --osd_scrub_sleep=3 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.121566/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-12-02T01:27:55.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-12-02T01:27:55.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-12-02T01:27:55.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-12-02T01:27:55.088 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:55.087+0000 7f62b8ab2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:55.090 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:55.089+0000 7f62b8ab2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:55.091 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:55.090+0000 7f62b8ab2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-12-02T01:27:55.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2023-12-02T01:27:55.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-12-02T01:27:55.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2023-12-02T01:27:55.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-12-02T01:27:55.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-12-02T01:27:55.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:55.333 INFO:tasks.workunit.client.0.smithi162.stdout:0 2023-12-02T01:27:55.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-12-02T01:27:55.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:55.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:27:55.616 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:55.615+0000 7f62b8ab2500 -1 Falling back to public interface 2023-12-02T01:27:55.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:56.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:56.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:56.653 INFO:tasks.workunit.client.0.smithi162.stdout:1 2023-12-02T01:27:56.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-12-02T01:27:56.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:56.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:27:56.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:57.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:57.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:57.940 INFO:tasks.workunit.client.0.smithi162.stdout:2 2023-12-02T01:27:57.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-12-02T01:27:57.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:57.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:27:58.002 INFO:tasks.workunit.client.0.smithi162.stderr:2023-12-02T01:27:58.000+0000 7f62b8ab2500 -1 osd.5 0 log_to_monitors true 2023-12-02T01:27:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-12-02T01:27:59.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-12-02T01:27:59.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-12-02T01:27:59.312 INFO:tasks.workunit.client.0.smithi162.stdout:3 2023-12-02T01:27:59.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-12-02T01:27:59.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-12-02T01:27:59.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-12-02T01:27:59.595 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1197368361,v1:127.0.0.1:6843/1197368361] [v2:127.0.0.1:6844/1197368361,v1:127.0.0.1:6845/1197368361] exists,up a01f5b57-1736-475b-bd61-7206afedb70b 2023-12-02T01:27:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-12-02T01:27:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-12-02T01:27:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-12-02T01:27:59.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: seq 1 3 2023-12-02T01:27:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2023-12-02T01:27:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:68: TEST_recover_unexpected: create_pool test1 16 16 2023-12-02T01:27:59.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 16 16 2023-12-02T01:27:59.989 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test1' created 2023-12-02T01:28:00.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-12-02T01:28:01.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2023-12-02T01:28:01.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:68: TEST_recover_unexpected: create_pool test2 16 16 2023-12-02T01:28:01.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 16 16 2023-12-02T01:28:02.022 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test2' created 2023-12-02T01:28:02.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-12-02T01:28:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2023-12-02T01:28:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:68: TEST_recover_unexpected: create_pool test3 16 16 2023-12-02T01:28:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 16 16 2023-12-02T01:28:04.112 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test3' created 2023-12-02T01:28:04.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-12-02T01:28:05.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:71: TEST_recover_unexpected: wait_for_clean 2023-12-02T01:28:05.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-12-02T01:28:05.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-12-02T01:28:05.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-12-02T01:28:05.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-12-02T01:28:05.129 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-12-02T01:28:05.129 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-12-02T01:28:05.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-12-02T01:28:05.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-12-02T01:28:05.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-12-02T01:28:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-12-02T01:28:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-12-02T01:28:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-12-02T01:28:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-12-02T01:28:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2023-12-02T01:28:05.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2023-12-02T01:28:05.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr:1 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr:2 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr:3 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr:4 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr:5' 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:28:05.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-12-02T01:28:05.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2023-12-02T01:28:05.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2023-12-02T01:28:05.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2023-12-02T01:28:05.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:28:05.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-12-02T01:28:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2023-12-02T01:28:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2023-12-02T01:28:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2023-12-02T01:28:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:28:06.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-12-02T01:28:06.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2023-12-02T01:28:06.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2023-12-02T01:28:06.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509450' 2023-12-02T01:28:06.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:28:06.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-12-02T01:28:06.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2023-12-02T01:28:06.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2023-12-02T01:28:06.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509450 3-85899345927' 2023-12-02T01:28:06.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:28:06.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-12-02T01:28:06.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2023-12-02T01:28:06.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2023-12-02T01:28:06.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509450 3-85899345927 4-107374182405' 2023-12-02T01:28:06.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2023-12-02T01:28:06.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2023-12-02T01:28:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018883 2023-12-02T01:28:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018883 2023-12-02T01:28:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509450 3-85899345927 4-107374182405 5-128849018883' 2023-12-02T01:28:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:28:06.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2023-12-02T01:28:06.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:28:06.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2023-12-02T01:28:06.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2023-12-02T01:28:06.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:28:06.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2023-12-02T01:28:06.486 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836494 2023-12-02T01:28:06.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2023-12-02T01:28:06.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2023-12-02T01:28:06.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2023-12-02T01:28:06.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2023-12-02T01:28:07.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2023-12-02T01:28:07.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2023-12-02T01:28:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836494 2023-12-02T01:28:08.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:28:08.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2023-12-02T01:28:08.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:28:08.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2023-12-02T01:28:08.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2023-12-02T01:28:08.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:28:08.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2023-12-02T01:28:08.071 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672972 2023-12-02T01:28:08.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2023-12-02T01:28:08.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2023-12-02T01:28:08.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2023-12-02T01:28:08.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:28:08.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2023-12-02T01:28:08.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:28:08.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2023-12-02T01:28:08.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2023-12-02T01:28:08.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:28:08.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2023-12-02T01:28:08.351 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509450 2023-12-02T01:28:08.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2023-12-02T01:28:08.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2023-12-02T01:28:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2023-12-02T01:28:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:28:08.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345927 2023-12-02T01:28:08.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:28:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2023-12-02T01:28:08.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345927 2023-12-02T01:28:08.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:28:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2023-12-02T01:28:08.638 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345927 2023-12-02T01:28:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2023-12-02T01:28:08.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2023-12-02T01:28:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345927 2023-12-02T01:28:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:28:08.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182405 2023-12-02T01:28:08.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:28:08.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2023-12-02T01:28:08.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182405 2023-12-02T01:28:08.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:28:08.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2023-12-02T01:28:08.959 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182405 2023-12-02T01:28:08.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2023-12-02T01:28:08.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2023-12-02T01:28:09.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2023-12-02T01:28:09.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2023-12-02T01:28:09.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018883 2023-12-02T01:28:09.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2023-12-02T01:28:09.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2023-12-02T01:28:09.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018883 2023-12-02T01:28:09.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2023-12-02T01:28:09.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018883 2023-12-02T01:28:09.279 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018883 2023-12-02T01:28:09.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2023-12-02T01:28:09.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2023-12-02T01:28:09.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018883 -lt 128849018883 2023-12-02T01:28:09.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-12-02T01:28:09.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-12-02T01:28:09.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-12-02T01:28:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 48 == 0 2023-12-02T01:28:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-12-02T01:28:09.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-12-02T01:28:09.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-12-02T01:28:09.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-12-02T01:28:09.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-12-02T01:28:09.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-12-02T01:28:09.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-12-02T01:28:10.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=48 2023-12-02T01:28:10.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-12-02T01:28:10.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-12-02T01:28:10.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-12-02T01:28:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 48 = 48 2023-12-02T01:28:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-12-02T01:28:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-12-02T01:28:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:73: TEST_recover_unexpected: dd if=/dev/urandom of=datafile bs=4k count=2 2023-12-02T01:28:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records in 2023-12-02T01:28:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records out 2023-12-02T01:28:10.516 INFO:tasks.workunit.client.0.smithi162.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 6.386e-05 s, 128 MB/s 2023-12-02T01:28:10.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 3 2023-12-02T01:28:10.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2023-12-02T01:28:10.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: seq 1 1000 2023-12-02T01:28:10.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:10.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj1 datafile 2023-12-02T01:28:10.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:10.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj2 datafile 2023-12-02T01:28:10.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:10.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj3 datafile 2023-12-02T01:28:10.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:10.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj4 datafile 2023-12-02T01:28:11.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:11.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj5 datafile 2023-12-02T01:28:11.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:11.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj6 datafile 2023-12-02T01:28:11.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:11.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj7 datafile 2023-12-02T01:28:11.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:11.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj8 datafile 2023-12-02T01:28:11.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:11.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj9 datafile 2023-12-02T01:28:11.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:11.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj10 datafile 2023-12-02T01:28:12.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:12.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj11 datafile 2023-12-02T01:28:12.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:12.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj12 datafile 2023-12-02T01:28:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj13 datafile 2023-12-02T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj14 datafile 2023-12-02T01:28:12.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:12.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj15 datafile 2023-12-02T01:28:12.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:12.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj16 datafile 2023-12-02T01:28:12.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:12.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj17 datafile 2023-12-02T01:28:13.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:13.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj18 datafile 2023-12-02T01:28:13.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj19 datafile 2023-12-02T01:28:13.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:13.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj20 datafile 2023-12-02T01:28:13.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:13.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj21 datafile 2023-12-02T01:28:13.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:13.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj22 datafile 2023-12-02T01:28:13.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:13.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj23 datafile 2023-12-02T01:28:13.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:13.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj24 datafile 2023-12-02T01:28:14.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:14.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj25 datafile 2023-12-02T01:28:14.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:14.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj26 datafile 2023-12-02T01:28:14.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:14.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj27 datafile 2023-12-02T01:28:14.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:14.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj28 datafile 2023-12-02T01:28:14.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:14.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj29 datafile 2023-12-02T01:28:14.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:14.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj30 datafile 2023-12-02T01:28:15.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:15.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj31 datafile 2023-12-02T01:28:15.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:15.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj32 datafile 2023-12-02T01:28:15.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:15.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj33 datafile 2023-12-02T01:28:15.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:15.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj34 datafile 2023-12-02T01:28:15.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:15.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj35 datafile 2023-12-02T01:28:15.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:15.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj36 datafile 2023-12-02T01:28:15.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:15.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj37 datafile 2023-12-02T01:28:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj38 datafile 2023-12-02T01:28:16.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:16.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj39 datafile 2023-12-02T01:28:16.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:16.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj40 datafile 2023-12-02T01:28:16.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:16.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj41 datafile 2023-12-02T01:28:16.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:16.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj42 datafile 2023-12-02T01:28:16.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:16.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj43 datafile 2023-12-02T01:28:17.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:17.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj44 datafile 2023-12-02T01:28:17.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:17.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj45 datafile 2023-12-02T01:28:17.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:17.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj46 datafile 2023-12-02T01:28:17.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:17.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj47 datafile 2023-12-02T01:28:17.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:17.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj48 datafile 2023-12-02T01:28:17.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:17.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj49 datafile 2023-12-02T01:28:17.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:17.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj50 datafile 2023-12-02T01:28:18.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:18.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj51 datafile 2023-12-02T01:28:18.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:18.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj52 datafile 2023-12-02T01:28:18.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:18.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj53 datafile 2023-12-02T01:28:18.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:18.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj54 datafile 2023-12-02T01:28:18.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:18.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj55 datafile 2023-12-02T01:28:18.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:18.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj56 datafile 2023-12-02T01:28:19.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:19.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj57 datafile 2023-12-02T01:28:19.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:19.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj58 datafile 2023-12-02T01:28:19.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:19.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj59 datafile 2023-12-02T01:28:19.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:19.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj60 datafile 2023-12-02T01:28:19.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:19.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj61 datafile 2023-12-02T01:28:19.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:19.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj62 datafile 2023-12-02T01:28:20.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:20.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj63 datafile 2023-12-02T01:28:20.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:20.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj64 datafile 2023-12-02T01:28:20.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:20.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj65 datafile 2023-12-02T01:28:20.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:20.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj66 datafile 2023-12-02T01:28:20.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:20.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj67 datafile 2023-12-02T01:28:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj68 datafile 2023-12-02T01:28:20.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:20.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj69 datafile 2023-12-02T01:28:21.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:21.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj70 datafile 2023-12-02T01:28:21.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:21.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj71 datafile 2023-12-02T01:28:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj72 datafile 2023-12-02T01:28:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj73 datafile 2023-12-02T01:28:21.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:21.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj74 datafile 2023-12-02T01:28:21.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:21.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj75 datafile 2023-12-02T01:28:21.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:21.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj76 datafile 2023-12-02T01:28:22.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:22.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj77 datafile 2023-12-02T01:28:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj78 datafile 2023-12-02T01:28:22.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:22.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj79 datafile 2023-12-02T01:28:22.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:22.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj80 datafile 2023-12-02T01:28:22.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:22.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj81 datafile 2023-12-02T01:28:22.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:22.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj82 datafile 2023-12-02T01:28:23.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:23.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj83 datafile 2023-12-02T01:28:23.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:23.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj84 datafile 2023-12-02T01:28:23.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:23.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj85 datafile 2023-12-02T01:28:23.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:23.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj86 datafile 2023-12-02T01:28:23.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:23.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj87 datafile 2023-12-02T01:28:23.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:23.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj88 datafile 2023-12-02T01:28:23.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:23.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj89 datafile 2023-12-02T01:28:24.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:24.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj90 datafile 2023-12-02T01:28:24.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:24.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj91 datafile 2023-12-02T01:28:24.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:24.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj92 datafile 2023-12-02T01:28:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj93 datafile 2023-12-02T01:28:24.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:24.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj94 datafile 2023-12-02T01:28:25.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:25.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj95 datafile 2023-12-02T01:28:25.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:25.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj96 datafile 2023-12-02T01:28:25.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:25.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj97 datafile 2023-12-02T01:28:25.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:25.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj98 datafile 2023-12-02T01:28:25.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:25.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj99 datafile 2023-12-02T01:28:25.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:25.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj100 datafile 2023-12-02T01:28:25.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:25.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj101 datafile 2023-12-02T01:28:26.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:26.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj102 datafile 2023-12-02T01:28:26.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:26.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj103 datafile 2023-12-02T01:28:26.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:26.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj104 datafile 2023-12-02T01:28:26.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:26.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj105 datafile 2023-12-02T01:28:26.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:26.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj106 datafile 2023-12-02T01:28:26.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:26.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj107 datafile 2023-12-02T01:28:26.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:26.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj108 datafile 2023-12-02T01:28:27.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:27.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj109 datafile 2023-12-02T01:28:27.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:27.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj110 datafile 2023-12-02T01:28:27.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:27.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj111 datafile 2023-12-02T01:28:27.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:27.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj112 datafile 2023-12-02T01:28:27.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:27.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj113 datafile 2023-12-02T01:28:27.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:27.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj114 datafile 2023-12-02T01:28:27.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:27.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj115 datafile 2023-12-02T01:28:28.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:28.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj116 datafile 2023-12-02T01:28:28.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:28.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj117 datafile 2023-12-02T01:28:28.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:28.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj118 datafile 2023-12-02T01:28:28.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:28.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj119 datafile 2023-12-02T01:28:28.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:28.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj120 datafile 2023-12-02T01:28:28.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:28.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj121 datafile 2023-12-02T01:28:29.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:29.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj122 datafile 2023-12-02T01:28:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj123 datafile 2023-12-02T01:28:29.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:29.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj124 datafile 2023-12-02T01:28:29.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:29.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj125 datafile 2023-12-02T01:28:29.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:29.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj126 datafile 2023-12-02T01:28:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj127 datafile 2023-12-02T01:28:29.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:29.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj128 datafile 2023-12-02T01:28:30.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:30.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj129 datafile 2023-12-02T01:28:30.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:30.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj130 datafile 2023-12-02T01:28:30.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:30.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj131 datafile 2023-12-02T01:28:30.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:30.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj132 datafile 2023-12-02T01:28:30.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:30.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj133 datafile 2023-12-02T01:28:30.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:30.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj134 datafile 2023-12-02T01:28:30.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:30.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj135 datafile 2023-12-02T01:28:31.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:31.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj136 datafile 2023-12-02T01:28:31.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:31.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj137 datafile 2023-12-02T01:28:31.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:31.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj138 datafile 2023-12-02T01:28:31.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:31.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj139 datafile 2023-12-02T01:28:31.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:31.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj140 datafile 2023-12-02T01:28:31.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:31.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj141 datafile 2023-12-02T01:28:31.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:31.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj142 datafile 2023-12-02T01:28:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:32.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj143 datafile 2023-12-02T01:28:32.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:32.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj144 datafile 2023-12-02T01:28:32.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:32.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj145 datafile 2023-12-02T01:28:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj146 datafile 2023-12-02T01:28:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj147 datafile 2023-12-02T01:28:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj148 datafile 2023-12-02T01:28:33.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:33.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj149 datafile 2023-12-02T01:28:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj150 datafile 2023-12-02T01:28:33.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:33.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj151 datafile 2023-12-02T01:28:33.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:33.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj152 datafile 2023-12-02T01:28:33.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:33.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj153 datafile 2023-12-02T01:28:33.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:33.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj154 datafile 2023-12-02T01:28:33.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:33.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj155 datafile 2023-12-02T01:28:34.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:34.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj156 datafile 2023-12-02T01:28:34.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:34.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj157 datafile 2023-12-02T01:28:34.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:34.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj158 datafile 2023-12-02T01:28:34.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:34.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj159 datafile 2023-12-02T01:28:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj160 datafile 2023-12-02T01:28:34.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:34.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj161 datafile 2023-12-02T01:28:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj162 datafile 2023-12-02T01:28:35.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:35.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj163 datafile 2023-12-02T01:28:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj164 datafile 2023-12-02T01:28:35.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:35.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj165 datafile 2023-12-02T01:28:35.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:35.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj166 datafile 2023-12-02T01:28:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj167 datafile 2023-12-02T01:28:35.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:35.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj168 datafile 2023-12-02T01:28:36.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:36.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj169 datafile 2023-12-02T01:28:36.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:36.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj170 datafile 2023-12-02T01:28:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj171 datafile 2023-12-02T01:28:36.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:36.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj172 datafile 2023-12-02T01:28:36.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:36.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj173 datafile 2023-12-02T01:28:36.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:36.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj174 datafile 2023-12-02T01:28:36.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:36.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj175 datafile 2023-12-02T01:28:37.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:37.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj176 datafile 2023-12-02T01:28:37.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:37.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj177 datafile 2023-12-02T01:28:37.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:37.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj178 datafile 2023-12-02T01:28:37.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:37.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj179 datafile 2023-12-02T01:28:37.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:37.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj180 datafile 2023-12-02T01:28:37.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:37.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj181 datafile 2023-12-02T01:28:37.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:37.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj182 datafile 2023-12-02T01:28:38.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:38.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj183 datafile 2023-12-02T01:28:38.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:38.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj184 datafile 2023-12-02T01:28:38.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:38.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj185 datafile 2023-12-02T01:28:38.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:38.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj186 datafile 2023-12-02T01:28:38.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:38.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj187 datafile 2023-12-02T01:28:38.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:38.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj188 datafile 2023-12-02T01:28:38.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:38.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj189 datafile 2023-12-02T01:28:39.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:39.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj190 datafile 2023-12-02T01:28:39.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj191 datafile 2023-12-02T01:28:39.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:39.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj192 datafile 2023-12-02T01:28:39.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:39.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj193 datafile 2023-12-02T01:28:39.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:39.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj194 datafile 2023-12-02T01:28:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj195 datafile 2023-12-02T01:28:39.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:39.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj196 datafile 2023-12-02T01:28:40.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:40.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj197 datafile 2023-12-02T01:28:40.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:40.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj198 datafile 2023-12-02T01:28:40.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:40.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj199 datafile 2023-12-02T01:28:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj200 datafile 2023-12-02T01:28:40.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:40.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj201 datafile 2023-12-02T01:28:40.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:40.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj202 datafile 2023-12-02T01:28:41.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:41.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj203 datafile 2023-12-02T01:28:41.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:41.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj204 datafile 2023-12-02T01:28:41.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:41.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj205 datafile 2023-12-02T01:28:41.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:41.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj206 datafile 2023-12-02T01:28:41.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:41.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj207 datafile 2023-12-02T01:28:41.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:41.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj208 datafile 2023-12-02T01:28:41.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:41.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj209 datafile 2023-12-02T01:28:42.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:42.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj210 datafile 2023-12-02T01:28:42.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:42.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj211 datafile 2023-12-02T01:28:42.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:42.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj212 datafile 2023-12-02T01:28:42.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:42.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj213 datafile 2023-12-02T01:28:42.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:42.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj214 datafile 2023-12-02T01:28:42.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:42.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj215 datafile 2023-12-02T01:28:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj216 datafile 2023-12-02T01:28:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj217 datafile 2023-12-02T01:28:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj218 datafile 2023-12-02T01:28:43.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:43.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj219 datafile 2023-12-02T01:28:43.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:43.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj220 datafile 2023-12-02T01:28:43.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:43.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj221 datafile 2023-12-02T01:28:43.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:43.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj222 datafile 2023-12-02T01:28:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:43.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj223 datafile 2023-12-02T01:28:44.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:44.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj224 datafile 2023-12-02T01:28:44.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:44.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj225 datafile 2023-12-02T01:28:44.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:44.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj226 datafile 2023-12-02T01:28:44.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:44.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj227 datafile 2023-12-02T01:28:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj228 datafile 2023-12-02T01:28:44.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:44.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj229 datafile 2023-12-02T01:28:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj230 datafile 2023-12-02T01:28:45.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:45.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj231 datafile 2023-12-02T01:28:45.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:45.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj232 datafile 2023-12-02T01:28:45.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:45.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj233 datafile 2023-12-02T01:28:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj234 datafile 2023-12-02T01:28:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj235 datafile 2023-12-02T01:28:45.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:45.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj236 datafile 2023-12-02T01:28:46.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:46.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj237 datafile 2023-12-02T01:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj238 datafile 2023-12-02T01:28:46.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:46.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj239 datafile 2023-12-02T01:28:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj240 datafile 2023-12-02T01:28:46.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:46.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj241 datafile 2023-12-02T01:28:46.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:46.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj242 datafile 2023-12-02T01:28:46.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:46.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj243 datafile 2023-12-02T01:28:47.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:47.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj244 datafile 2023-12-02T01:28:47.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:47.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj245 datafile 2023-12-02T01:28:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:47.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj246 datafile 2023-12-02T01:28:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj247 datafile 2023-12-02T01:28:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:47.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj248 datafile 2023-12-02T01:28:47.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:47.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj249 datafile 2023-12-02T01:28:47.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:47.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj250 datafile 2023-12-02T01:28:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:48.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj251 datafile 2023-12-02T01:28:48.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:48.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj252 datafile 2023-12-02T01:28:48.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:48.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj253 datafile 2023-12-02T01:28:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj254 datafile 2023-12-02T01:28:48.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:48.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj255 datafile 2023-12-02T01:28:48.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:48.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj256 datafile 2023-12-02T01:28:48.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:48.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj257 datafile 2023-12-02T01:28:49.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:49.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj258 datafile 2023-12-02T01:28:49.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:49.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj259 datafile 2023-12-02T01:28:49.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:49.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj260 datafile 2023-12-02T01:28:49.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:49.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj261 datafile 2023-12-02T01:28:49.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:49.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj262 datafile 2023-12-02T01:28:49.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:49.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj263 datafile 2023-12-02T01:28:50.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:50.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj264 datafile 2023-12-02T01:28:50.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:50.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj265 datafile 2023-12-02T01:28:50.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:50.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj266 datafile 2023-12-02T01:28:50.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:50.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj267 datafile 2023-12-02T01:28:50.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:50.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj268 datafile 2023-12-02T01:28:50.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:50.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj269 datafile 2023-12-02T01:28:50.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:50.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj270 datafile 2023-12-02T01:28:51.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:51.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj271 datafile 2023-12-02T01:28:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj272 datafile 2023-12-02T01:28:51.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:51.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj273 datafile 2023-12-02T01:28:51.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:51.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj274 datafile 2023-12-02T01:28:51.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:51.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj275 datafile 2023-12-02T01:28:51.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:51.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj276 datafile 2023-12-02T01:28:51.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:51.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj277 datafile 2023-12-02T01:28:52.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:52.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj278 datafile 2023-12-02T01:28:52.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:52.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj279 datafile 2023-12-02T01:28:52.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:52.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj280 datafile 2023-12-02T01:28:52.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:52.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj281 datafile 2023-12-02T01:28:52.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:52.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj282 datafile 2023-12-02T01:28:52.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:52.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj283 datafile 2023-12-02T01:28:53.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:53.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj284 datafile 2023-12-02T01:28:53.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:53.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj285 datafile 2023-12-02T01:28:53.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:53.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj286 datafile 2023-12-02T01:28:53.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:53.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj287 datafile 2023-12-02T01:28:53.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:53.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj288 datafile 2023-12-02T01:28:53.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:53.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj289 datafile 2023-12-02T01:28:53.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:53.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj290 datafile 2023-12-02T01:28:54.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:54.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj291 datafile 2023-12-02T01:28:54.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:54.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj292 datafile 2023-12-02T01:28:54.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:54.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj293 datafile 2023-12-02T01:28:54.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:54.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj294 datafile 2023-12-02T01:28:54.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:54.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj295 datafile 2023-12-02T01:28:54.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:54.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj296 datafile 2023-12-02T01:28:54.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:54.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj297 datafile 2023-12-02T01:28:55.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:55.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj298 datafile 2023-12-02T01:28:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj299 datafile 2023-12-02T01:28:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj300 datafile 2023-12-02T01:28:55.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:55.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj301 datafile 2023-12-02T01:28:55.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:55.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj302 datafile 2023-12-02T01:28:55.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:55.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj303 datafile 2023-12-02T01:28:55.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:55.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj304 datafile 2023-12-02T01:28:56.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:56.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj305 datafile 2023-12-02T01:28:56.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:56.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj306 datafile 2023-12-02T01:28:56.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:56.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj307 datafile 2023-12-02T01:28:56.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:56.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj308 datafile 2023-12-02T01:28:56.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:56.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj309 datafile 2023-12-02T01:28:57.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:57.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj310 datafile 2023-12-02T01:28:57.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:57.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj311 datafile 2023-12-02T01:28:57.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:57.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj312 datafile 2023-12-02T01:28:57.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:57.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj313 datafile 2023-12-02T01:28:57.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:57.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj314 datafile 2023-12-02T01:28:57.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:57.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj315 datafile 2023-12-02T01:28:57.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:57.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj316 datafile 2023-12-02T01:28:58.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:58.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj317 datafile 2023-12-02T01:28:58.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:58.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj318 datafile 2023-12-02T01:28:58.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:58.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj319 datafile 2023-12-02T01:28:58.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:58.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj320 datafile 2023-12-02T01:28:58.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:58.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj321 datafile 2023-12-02T01:28:58.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:58.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj322 datafile 2023-12-02T01:28:58.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:58.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj323 datafile 2023-12-02T01:28:59.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:59.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj324 datafile 2023-12-02T01:28:59.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:59.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj325 datafile 2023-12-02T01:28:59.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:59.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj326 datafile 2023-12-02T01:28:59.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:59.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj327 datafile 2023-12-02T01:28:59.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:59.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj328 datafile 2023-12-02T01:28:59.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:59.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj329 datafile 2023-12-02T01:28:59.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:28:59.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj330 datafile 2023-12-02T01:29:00.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:00.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj331 datafile 2023-12-02T01:29:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj332 datafile 2023-12-02T01:29:00.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:00.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj333 datafile 2023-12-02T01:29:00.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:00.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj334 datafile 2023-12-02T01:29:00.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:00.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj335 datafile 2023-12-02T01:29:00.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:00.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj336 datafile 2023-12-02T01:29:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj337 datafile 2023-12-02T01:29:01.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:01.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj338 datafile 2023-12-02T01:29:01.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:01.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj339 datafile 2023-12-02T01:29:01.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:01.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj340 datafile 2023-12-02T01:29:01.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:01.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj341 datafile 2023-12-02T01:29:01.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:01.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj342 datafile 2023-12-02T01:29:01.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:01.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj343 datafile 2023-12-02T01:29:02.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:02.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj344 datafile 2023-12-02T01:29:02.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:02.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj345 datafile 2023-12-02T01:29:02.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:02.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj346 datafile 2023-12-02T01:29:02.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:02.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj347 datafile 2023-12-02T01:29:02.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:02.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj348 datafile 2023-12-02T01:29:02.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:02.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj349 datafile 2023-12-02T01:29:02.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:02.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj350 datafile 2023-12-02T01:29:03.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:03.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj351 datafile 2023-12-02T01:29:03.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:03.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj352 datafile 2023-12-02T01:29:03.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:03.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj353 datafile 2023-12-02T01:29:03.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:03.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj354 datafile 2023-12-02T01:29:03.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:03.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj355 datafile 2023-12-02T01:29:03.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:03.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj356 datafile 2023-12-02T01:29:04.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:04.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj357 datafile 2023-12-02T01:29:04.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:04.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj358 datafile 2023-12-02T01:29:04.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:04.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj359 datafile 2023-12-02T01:29:04.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:04.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj360 datafile 2023-12-02T01:29:04.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:04.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj361 datafile 2023-12-02T01:29:04.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:04.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj362 datafile 2023-12-02T01:29:04.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:04.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj363 datafile 2023-12-02T01:29:05.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:05.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj364 datafile 2023-12-02T01:29:05.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:05.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj365 datafile 2023-12-02T01:29:05.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:05.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj366 datafile 2023-12-02T01:29:05.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:05.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj367 datafile 2023-12-02T01:29:05.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:05.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj368 datafile 2023-12-02T01:29:05.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:05.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj369 datafile 2023-12-02T01:29:05.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:05.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj370 datafile 2023-12-02T01:29:06.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:06.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj371 datafile 2023-12-02T01:29:06.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:06.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj372 datafile 2023-12-02T01:29:06.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:06.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj373 datafile 2023-12-02T01:29:06.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:06.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj374 datafile 2023-12-02T01:29:06.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:06.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj375 datafile 2023-12-02T01:29:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj376 datafile 2023-12-02T01:29:07.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:07.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj377 datafile 2023-12-02T01:29:07.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:07.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj378 datafile 2023-12-02T01:29:07.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:07.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj379 datafile 2023-12-02T01:29:07.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:07.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj380 datafile 2023-12-02T01:29:07.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:07.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj381 datafile 2023-12-02T01:29:07.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:07.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj382 datafile 2023-12-02T01:29:07.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:07.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj383 datafile 2023-12-02T01:29:08.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:08.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj384 datafile 2023-12-02T01:29:08.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:08.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj385 datafile 2023-12-02T01:29:08.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:08.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj386 datafile 2023-12-02T01:29:08.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:08.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj387 datafile 2023-12-02T01:29:08.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:08.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj388 datafile 2023-12-02T01:29:08.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:08.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj389 datafile 2023-12-02T01:29:08.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:08.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj390 datafile 2023-12-02T01:29:09.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:09.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj391 datafile 2023-12-02T01:29:09.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:09.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj392 datafile 2023-12-02T01:29:09.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:09.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj393 datafile 2023-12-02T01:29:09.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:09.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj394 datafile 2023-12-02T01:29:09.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:09.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj395 datafile 2023-12-02T01:29:09.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:09.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj396 datafile 2023-12-02T01:29:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj397 datafile 2023-12-02T01:29:10.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:10.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj398 datafile 2023-12-02T01:29:10.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:10.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj399 datafile 2023-12-02T01:29:10.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:10.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj400 datafile 2023-12-02T01:29:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj401 datafile 2023-12-02T01:29:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj402 datafile 2023-12-02T01:29:10.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:10.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj403 datafile 2023-12-02T01:29:10.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:10.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj404 datafile 2023-12-02T01:29:11.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:11.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj405 datafile 2023-12-02T01:29:11.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:11.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj406 datafile 2023-12-02T01:29:11.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:11.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj407 datafile 2023-12-02T01:29:11.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:11.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj408 datafile 2023-12-02T01:29:11.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:11.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj409 datafile 2023-12-02T01:29:11.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:11.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj410 datafile 2023-12-02T01:29:12.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:12.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj411 datafile 2023-12-02T01:29:12.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:12.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj412 datafile 2023-12-02T01:29:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj413 datafile 2023-12-02T01:29:12.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:12.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj414 datafile 2023-12-02T01:29:12.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:12.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj415 datafile 2023-12-02T01:29:12.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:12.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj416 datafile 2023-12-02T01:29:12.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:12.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj417 datafile 2023-12-02T01:29:13.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:13.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj418 datafile 2023-12-02T01:29:13.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:13.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj419 datafile 2023-12-02T01:29:13.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:13.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj420 datafile 2023-12-02T01:29:13.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:13.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj421 datafile 2023-12-02T01:29:13.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:13.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj422 datafile 2023-12-02T01:29:13.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:13.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj423 datafile 2023-12-02T01:29:13.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:13.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj424 datafile 2023-12-02T01:29:14.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:14.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj425 datafile 2023-12-02T01:29:14.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:14.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj426 datafile 2023-12-02T01:29:14.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:14.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj427 datafile 2023-12-02T01:29:14.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:14.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj428 datafile 2023-12-02T01:29:14.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:14.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj429 datafile 2023-12-02T01:29:14.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:14.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj430 datafile 2023-12-02T01:29:15.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:15.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj431 datafile 2023-12-02T01:29:15.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:15.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj432 datafile 2023-12-02T01:29:15.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:15.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj433 datafile 2023-12-02T01:29:15.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:15.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj434 datafile 2023-12-02T01:29:15.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:15.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj435 datafile 2023-12-02T01:29:15.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:15.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj436 datafile 2023-12-02T01:29:15.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:15.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj437 datafile 2023-12-02T01:29:16.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:16.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj438 datafile 2023-12-02T01:29:16.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:16.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj439 datafile 2023-12-02T01:29:16.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:16.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj440 datafile 2023-12-02T01:29:16.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:16.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj441 datafile 2023-12-02T01:29:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj442 datafile 2023-12-02T01:29:16.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:16.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj443 datafile 2023-12-02T01:29:16.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:16.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj444 datafile 2023-12-02T01:29:17.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:17.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj445 datafile 2023-12-02T01:29:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj446 datafile 2023-12-02T01:29:17.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:17.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj447 datafile 2023-12-02T01:29:17.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:17.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj448 datafile 2023-12-02T01:29:17.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:17.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj449 datafile 2023-12-02T01:29:17.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:17.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj450 datafile 2023-12-02T01:29:17.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:17.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj451 datafile 2023-12-02T01:29:18.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:18.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj452 datafile 2023-12-02T01:29:18.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:18.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj453 datafile 2023-12-02T01:29:18.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:18.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj454 datafile 2023-12-02T01:29:18.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:18.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj455 datafile 2023-12-02T01:29:18.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:18.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj456 datafile 2023-12-02T01:29:18.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:18.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj457 datafile 2023-12-02T01:29:19.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:19.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj458 datafile 2023-12-02T01:29:19.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:19.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj459 datafile 2023-12-02T01:29:19.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:19.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj460 datafile 2023-12-02T01:29:19.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:19.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj461 datafile 2023-12-02T01:29:19.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:19.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj462 datafile 2023-12-02T01:29:19.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:19.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj463 datafile 2023-12-02T01:29:19.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:19.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj464 datafile 2023-12-02T01:29:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:20.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj465 datafile 2023-12-02T01:29:20.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:20.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj466 datafile 2023-12-02T01:29:20.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:20.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj467 datafile 2023-12-02T01:29:20.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:20.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj468 datafile 2023-12-02T01:29:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj469 datafile 2023-12-02T01:29:20.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:20.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj470 datafile 2023-12-02T01:29:20.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:20.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj471 datafile 2023-12-02T01:29:21.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:21.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj472 datafile 2023-12-02T01:29:21.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:21.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj473 datafile 2023-12-02T01:29:21.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:21.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj474 datafile 2023-12-02T01:29:21.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:21.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj475 datafile 2023-12-02T01:29:21.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:21.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj476 datafile 2023-12-02T01:29:21.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:21.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj477 datafile 2023-12-02T01:29:22.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:22.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj478 datafile 2023-12-02T01:29:22.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:22.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj479 datafile 2023-12-02T01:29:22.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:22.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj480 datafile 2023-12-02T01:29:22.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:22.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj481 datafile 2023-12-02T01:29:22.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:22.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj482 datafile 2023-12-02T01:29:22.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:22.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj483 datafile 2023-12-02T01:29:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj484 datafile 2023-12-02T01:29:23.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:23.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj485 datafile 2023-12-02T01:29:23.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:23.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj486 datafile 2023-12-02T01:29:23.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:23.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj487 datafile 2023-12-02T01:29:23.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:23.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj488 datafile 2023-12-02T01:29:23.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:23.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj489 datafile 2023-12-02T01:29:24.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:24.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj490 datafile 2023-12-02T01:29:24.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:24.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj491 datafile 2023-12-02T01:29:24.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:24.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj492 datafile 2023-12-02T01:29:24.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:24.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj493 datafile 2023-12-02T01:29:24.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:24.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj494 datafile 2023-12-02T01:29:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj495 datafile 2023-12-02T01:29:25.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:25.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj496 datafile 2023-12-02T01:29:25.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:25.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj497 datafile 2023-12-02T01:29:25.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:25.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj498 datafile 2023-12-02T01:29:25.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:25.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj499 datafile 2023-12-02T01:29:25.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:25.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj500 datafile 2023-12-02T01:29:25.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:25.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj501 datafile 2023-12-02T01:29:25.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:25.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj502 datafile 2023-12-02T01:29:26.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:26.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj503 datafile 2023-12-02T01:29:26.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:26.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj504 datafile 2023-12-02T01:29:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj505 datafile 2023-12-02T01:29:26.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:26.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj506 datafile 2023-12-02T01:29:26.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:26.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj507 datafile 2023-12-02T01:29:26.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:26.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj508 datafile 2023-12-02T01:29:27.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:27.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj509 datafile 2023-12-02T01:29:27.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:27.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj510 datafile 2023-12-02T01:29:27.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:27.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj511 datafile 2023-12-02T01:29:27.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:27.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj512 datafile 2023-12-02T01:29:27.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:27.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj513 datafile 2023-12-02T01:29:28.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:28.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj514 datafile 2023-12-02T01:29:28.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:28.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj515 datafile 2023-12-02T01:29:28.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:28.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj516 datafile 2023-12-02T01:29:28.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:28.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj517 datafile 2023-12-02T01:29:28.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:28.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj518 datafile 2023-12-02T01:29:28.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:28.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj519 datafile 2023-12-02T01:29:29.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj520 datafile 2023-12-02T01:29:29.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:29.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj521 datafile 2023-12-02T01:29:29.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:29.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj522 datafile 2023-12-02T01:29:29.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:29.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj523 datafile 2023-12-02T01:29:29.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:29.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj524 datafile 2023-12-02T01:29:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj525 datafile 2023-12-02T01:29:29.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:29.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj526 datafile 2023-12-02T01:29:30.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:30.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj527 datafile 2023-12-02T01:29:30.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:30.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj528 datafile 2023-12-02T01:29:30.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:30.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj529 datafile 2023-12-02T01:29:30.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:30.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj530 datafile 2023-12-02T01:29:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj531 datafile 2023-12-02T01:29:30.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:30.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj532 datafile 2023-12-02T01:29:31.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:31.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj533 datafile 2023-12-02T01:29:31.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:31.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj534 datafile 2023-12-02T01:29:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj535 datafile 2023-12-02T01:29:31.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj536 datafile 2023-12-02T01:29:31.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:31.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj537 datafile 2023-12-02T01:29:31.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:31.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj538 datafile 2023-12-02T01:29:31.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:31.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj539 datafile 2023-12-02T01:29:32.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:32.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj540 datafile 2023-12-02T01:29:32.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:32.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj541 datafile 2023-12-02T01:29:32.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:32.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj542 datafile 2023-12-02T01:29:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj543 datafile 2023-12-02T01:29:32.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:32.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj544 datafile 2023-12-02T01:29:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj545 datafile 2023-12-02T01:29:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj546 datafile 2023-12-02T01:29:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj547 datafile 2023-12-02T01:29:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj548 datafile 2023-12-02T01:29:33.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:33.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj549 datafile 2023-12-02T01:29:33.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:33.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj550 datafile 2023-12-02T01:29:33.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:33.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj551 datafile 2023-12-02T01:29:33.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:33.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj552 datafile 2023-12-02T01:29:34.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:34.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj553 datafile 2023-12-02T01:29:34.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:34.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj554 datafile 2023-12-02T01:29:34.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:34.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj555 datafile 2023-12-02T01:29:34.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:34.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj556 datafile 2023-12-02T01:29:34.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:34.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj557 datafile 2023-12-02T01:29:34.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:34.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj558 datafile 2023-12-02T01:29:35.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:35.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj559 datafile 2023-12-02T01:29:35.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:35.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj560 datafile 2023-12-02T01:29:35.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:35.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj561 datafile 2023-12-02T01:29:35.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:35.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj562 datafile 2023-12-02T01:29:35.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:35.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj563 datafile 2023-12-02T01:29:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj564 datafile 2023-12-02T01:29:35.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:35.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj565 datafile 2023-12-02T01:29:36.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:36.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj566 datafile 2023-12-02T01:29:36.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:36.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj567 datafile 2023-12-02T01:29:36.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:36.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj568 datafile 2023-12-02T01:29:36.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:36.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj569 datafile 2023-12-02T01:29:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj570 datafile 2023-12-02T01:29:36.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:36.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj571 datafile 2023-12-02T01:29:37.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:37.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj572 datafile 2023-12-02T01:29:37.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:37.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj573 datafile 2023-12-02T01:29:37.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:37.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj574 datafile 2023-12-02T01:29:37.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:37.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj575 datafile 2023-12-02T01:29:37.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:37.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj576 datafile 2023-12-02T01:29:37.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:37.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj577 datafile 2023-12-02T01:29:38.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:38.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj578 datafile 2023-12-02T01:29:38.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:38.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj579 datafile 2023-12-02T01:29:38.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:38.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj580 datafile 2023-12-02T01:29:38.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:38.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj581 datafile 2023-12-02T01:29:38.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:38.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj582 datafile 2023-12-02T01:29:38.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:38.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj583 datafile 2023-12-02T01:29:38.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:38.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj584 datafile 2023-12-02T01:29:39.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:39.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj585 datafile 2023-12-02T01:29:39.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:39.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj586 datafile 2023-12-02T01:29:39.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:39.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj587 datafile 2023-12-02T01:29:39.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:39.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj588 datafile 2023-12-02T01:29:39.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:39.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj589 datafile 2023-12-02T01:29:39.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:39.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj590 datafile 2023-12-02T01:29:39.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:39.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj591 datafile 2023-12-02T01:29:40.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:40.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj592 datafile 2023-12-02T01:29:40.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:40.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj593 datafile 2023-12-02T01:29:40.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:40.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj594 datafile 2023-12-02T01:29:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj595 datafile 2023-12-02T01:29:40.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:40.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj596 datafile 2023-12-02T01:29:40.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:40.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj597 datafile 2023-12-02T01:29:40.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:40.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj598 datafile 2023-12-02T01:29:41.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:41.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj599 datafile 2023-12-02T01:29:41.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:41.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj600 datafile 2023-12-02T01:29:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj601 datafile 2023-12-02T01:29:41.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj602 datafile 2023-12-02T01:29:41.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:41.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj603 datafile 2023-12-02T01:29:41.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:41.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj604 datafile 2023-12-02T01:29:41.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:41.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj605 datafile 2023-12-02T01:29:42.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:42.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj606 datafile 2023-12-02T01:29:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj607 datafile 2023-12-02T01:29:42.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:42.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj608 datafile 2023-12-02T01:29:42.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:42.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj609 datafile 2023-12-02T01:29:42.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:42.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj610 datafile 2023-12-02T01:29:42.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:42.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj611 datafile 2023-12-02T01:29:43.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:43.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj612 datafile 2023-12-02T01:29:43.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:43.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj613 datafile 2023-12-02T01:29:43.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:43.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj614 datafile 2023-12-02T01:29:43.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:43.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj615 datafile 2023-12-02T01:29:43.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:43.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj616 datafile 2023-12-02T01:29:43.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:43.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj617 datafile 2023-12-02T01:29:44.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:44.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj618 datafile 2023-12-02T01:29:44.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:44.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj619 datafile 2023-12-02T01:29:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj620 datafile 2023-12-02T01:29:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj621 datafile 2023-12-02T01:29:44.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj622 datafile 2023-12-02T01:29:44.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:44.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj623 datafile 2023-12-02T01:29:44.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:44.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj624 datafile 2023-12-02T01:29:45.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:45.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj625 datafile 2023-12-02T01:29:45.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:45.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj626 datafile 2023-12-02T01:29:45.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:45.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj627 datafile 2023-12-02T01:29:45.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:45.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj628 datafile 2023-12-02T01:29:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj629 datafile 2023-12-02T01:29:45.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:45.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj630 datafile 2023-12-02T01:29:46.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:46.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj631 datafile 2023-12-02T01:29:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj632 datafile 2023-12-02T01:29:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj633 datafile 2023-12-02T01:29:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj634 datafile 2023-12-02T01:29:46.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:46.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj635 datafile 2023-12-02T01:29:46.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:46.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj636 datafile 2023-12-02T01:29:46.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:46.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj637 datafile 2023-12-02T01:29:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj638 datafile 2023-12-02T01:29:47.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:47.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj639 datafile 2023-12-02T01:29:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj640 datafile 2023-12-02T01:29:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj641 datafile 2023-12-02T01:29:47.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:47.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj642 datafile 2023-12-02T01:29:47.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:47.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj643 datafile 2023-12-02T01:29:47.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:47.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj644 datafile 2023-12-02T01:29:48.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:48.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj645 datafile 2023-12-02T01:29:48.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:48.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj646 datafile 2023-12-02T01:29:48.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:48.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj647 datafile 2023-12-02T01:29:48.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:48.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj648 datafile 2023-12-02T01:29:48.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:48.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj649 datafile 2023-12-02T01:29:48.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:48.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj650 datafile 2023-12-02T01:29:49.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:49.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj651 datafile 2023-12-02T01:29:49.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:49.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj652 datafile 2023-12-02T01:29:49.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:49.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj653 datafile 2023-12-02T01:29:49.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:49.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj654 datafile 2023-12-02T01:29:49.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:49.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj655 datafile 2023-12-02T01:29:49.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:49.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj656 datafile 2023-12-02T01:29:49.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:49.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj657 datafile 2023-12-02T01:29:50.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:50.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj658 datafile 2023-12-02T01:29:50.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:50.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj659 datafile 2023-12-02T01:29:50.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:50.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj660 datafile 2023-12-02T01:29:50.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:50.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj661 datafile 2023-12-02T01:29:50.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:50.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj662 datafile 2023-12-02T01:29:50.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:50.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj663 datafile 2023-12-02T01:29:50.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:50.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj664 datafile 2023-12-02T01:29:51.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:51.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj665 datafile 2023-12-02T01:29:51.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:51.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj666 datafile 2023-12-02T01:29:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj667 datafile 2023-12-02T01:29:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj668 datafile 2023-12-02T01:29:51.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:51.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj669 datafile 2023-12-02T01:29:51.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:51.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj670 datafile 2023-12-02T01:29:52.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:52.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj671 datafile 2023-12-02T01:29:52.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:52.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj672 datafile 2023-12-02T01:29:52.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:52.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj673 datafile 2023-12-02T01:29:52.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:52.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj674 datafile 2023-12-02T01:29:52.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:52.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj675 datafile 2023-12-02T01:29:52.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:52.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj676 datafile 2023-12-02T01:29:53.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:53.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj677 datafile 2023-12-02T01:29:53.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:53.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj678 datafile 2023-12-02T01:29:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj679 datafile 2023-12-02T01:29:53.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:53.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj680 datafile 2023-12-02T01:29:53.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:53.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj681 datafile 2023-12-02T01:29:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj682 datafile 2023-12-02T01:29:53.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:53.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj683 datafile 2023-12-02T01:29:54.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:54.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj684 datafile 2023-12-02T01:29:54.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:54.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj685 datafile 2023-12-02T01:29:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj686 datafile 2023-12-02T01:29:54.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj687 datafile 2023-12-02T01:29:54.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:54.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj688 datafile 2023-12-02T01:29:54.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:54.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj689 datafile 2023-12-02T01:29:54.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:54.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj690 datafile 2023-12-02T01:29:55.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:55.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj691 datafile 2023-12-02T01:29:55.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:55.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj692 datafile 2023-12-02T01:29:55.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:55.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj693 datafile 2023-12-02T01:29:55.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:55.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj694 datafile 2023-12-02T01:29:55.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:55.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj695 datafile 2023-12-02T01:29:55.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:55.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj696 datafile 2023-12-02T01:29:55.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:55.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj697 datafile 2023-12-02T01:29:56.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:56.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj698 datafile 2023-12-02T01:29:56.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:56.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj699 datafile 2023-12-02T01:29:56.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:56.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj700 datafile 2023-12-02T01:29:56.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:56.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj701 datafile 2023-12-02T01:29:56.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:56.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj702 datafile 2023-12-02T01:29:56.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:56.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj703 datafile 2023-12-02T01:29:56.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:56.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj704 datafile 2023-12-02T01:29:57.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:57.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj705 datafile 2023-12-02T01:29:57.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:57.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj706 datafile 2023-12-02T01:29:57.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:57.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj707 datafile 2023-12-02T01:29:57.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:57.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj708 datafile 2023-12-02T01:29:57.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:57.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj709 datafile 2023-12-02T01:29:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj710 datafile 2023-12-02T01:29:58.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:58.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj711 datafile 2023-12-02T01:29:58.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:58.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj712 datafile 2023-12-02T01:29:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj713 datafile 2023-12-02T01:29:58.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:58.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj714 datafile 2023-12-02T01:29:58.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:58.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj715 datafile 2023-12-02T01:29:59.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:59.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj716 datafile 2023-12-02T01:29:59.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:59.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj717 datafile 2023-12-02T01:29:59.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:59.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj718 datafile 2023-12-02T01:29:59.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:59.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj719 datafile 2023-12-02T01:29:59.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:59.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj720 datafile 2023-12-02T01:29:59.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:29:59.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj721 datafile 2023-12-02T01:30:00.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:00.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj722 datafile 2023-12-02T01:30:00.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:00.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj723 datafile 2023-12-02T01:30:00.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj724 datafile 2023-12-02T01:30:00.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:00.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj725 datafile 2023-12-02T01:30:00.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:00.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj726 datafile 2023-12-02T01:30:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj727 datafile 2023-12-02T01:30:00.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:00.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj728 datafile 2023-12-02T01:30:01.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:01.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj729 datafile 2023-12-02T01:30:01.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:01.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj730 datafile 2023-12-02T01:30:01.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:01.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj731 datafile 2023-12-02T01:30:01.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:01.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj732 datafile 2023-12-02T01:30:01.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:01.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj733 datafile 2023-12-02T01:30:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj734 datafile 2023-12-02T01:30:01.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:01.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj735 datafile 2023-12-02T01:30:02.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:02.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj736 datafile 2023-12-02T01:30:02.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:02.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj737 datafile 2023-12-02T01:30:02.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:02.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj738 datafile 2023-12-02T01:30:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj739 datafile 2023-12-02T01:30:02.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:02.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj740 datafile 2023-12-02T01:30:02.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:02.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj741 datafile 2023-12-02T01:30:02.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:02.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj742 datafile 2023-12-02T01:30:03.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:03.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj743 datafile 2023-12-02T01:30:03.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:03.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj744 datafile 2023-12-02T01:30:03.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:03.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj745 datafile 2023-12-02T01:30:03.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:03.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj746 datafile 2023-12-02T01:30:03.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:03.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj747 datafile 2023-12-02T01:30:03.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:03.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj748 datafile 2023-12-02T01:30:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj749 datafile 2023-12-02T01:30:04.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:04.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj750 datafile 2023-12-02T01:30:04.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:04.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj751 datafile 2023-12-02T01:30:04.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:04.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj752 datafile 2023-12-02T01:30:04.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:04.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj753 datafile 2023-12-02T01:30:04.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:04.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj754 datafile 2023-12-02T01:30:04.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:04.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj755 datafile 2023-12-02T01:30:04.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:04.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj756 datafile 2023-12-02T01:30:05.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:05.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj757 datafile 2023-12-02T01:30:05.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:05.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj758 datafile 2023-12-02T01:30:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj759 datafile 2023-12-02T01:30:05.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:05.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj760 datafile 2023-12-02T01:30:05.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:05.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj761 datafile 2023-12-02T01:30:05.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:05.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj762 datafile 2023-12-02T01:30:06.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:06.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj763 datafile 2023-12-02T01:30:06.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:06.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj764 datafile 2023-12-02T01:30:06.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:06.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj765 datafile 2023-12-02T01:30:06.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:06.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj766 datafile 2023-12-02T01:30:06.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:06.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj767 datafile 2023-12-02T01:30:06.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:06.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj768 datafile 2023-12-02T01:30:06.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:06.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj769 datafile 2023-12-02T01:30:07.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:07.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj770 datafile 2023-12-02T01:30:07.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:07.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj771 datafile 2023-12-02T01:30:07.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:07.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj772 datafile 2023-12-02T01:30:07.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:07.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj773 datafile 2023-12-02T01:30:07.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:07.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj774 datafile 2023-12-02T01:30:07.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:07.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj775 datafile 2023-12-02T01:30:08.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:08.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj776 datafile 2023-12-02T01:30:08.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:08.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj777 datafile 2023-12-02T01:30:08.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:08.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj778 datafile 2023-12-02T01:30:08.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:08.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj779 datafile 2023-12-02T01:30:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj780 datafile 2023-12-02T01:30:08.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:08.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj781 datafile 2023-12-02T01:30:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj782 datafile 2023-12-02T01:30:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj783 datafile 2023-12-02T01:30:09.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:09.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj784 datafile 2023-12-02T01:30:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj785 datafile 2023-12-02T01:30:09.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:09.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj786 datafile 2023-12-02T01:30:09.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:09.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj787 datafile 2023-12-02T01:30:09.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:09.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj788 datafile 2023-12-02T01:30:10.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:10.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj789 datafile 2023-12-02T01:30:10.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:10.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj790 datafile 2023-12-02T01:30:10.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:10.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj791 datafile 2023-12-02T01:30:10.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:10.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj792 datafile 2023-12-02T01:30:10.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:10.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj793 datafile 2023-12-02T01:30:10.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:10.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj794 datafile 2023-12-02T01:30:11.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:11.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj795 datafile 2023-12-02T01:30:11.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:11.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj796 datafile 2023-12-02T01:30:11.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:11.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj797 datafile 2023-12-02T01:30:11.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:11.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj798 datafile 2023-12-02T01:30:11.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:11.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj799 datafile 2023-12-02T01:30:11.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:11.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj800 datafile 2023-12-02T01:30:11.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:11.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj801 datafile 2023-12-02T01:30:12.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:12.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj802 datafile 2023-12-02T01:30:12.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:12.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj803 datafile 2023-12-02T01:30:12.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:12.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj804 datafile 2023-12-02T01:30:12.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:12.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj805 datafile 2023-12-02T01:30:12.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:12.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj806 datafile 2023-12-02T01:30:12.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:12.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj807 datafile 2023-12-02T01:30:12.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj808 datafile 2023-12-02T01:30:13.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:13.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj809 datafile 2023-12-02T01:30:13.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:13.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj810 datafile 2023-12-02T01:30:13.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:13.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj811 datafile 2023-12-02T01:30:13.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:13.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj812 datafile 2023-12-02T01:30:13.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:13.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj813 datafile 2023-12-02T01:30:13.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:13.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj814 datafile 2023-12-02T01:30:14.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:14.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj815 datafile 2023-12-02T01:30:14.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:14.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj816 datafile 2023-12-02T01:30:14.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:14.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj817 datafile 2023-12-02T01:30:14.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:14.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj818 datafile 2023-12-02T01:30:14.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:14.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj819 datafile 2023-12-02T01:30:14.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:14.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj820 datafile 2023-12-02T01:30:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj821 datafile 2023-12-02T01:30:15.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:15.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj822 datafile 2023-12-02T01:30:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj823 datafile 2023-12-02T01:30:15.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:15.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj824 datafile 2023-12-02T01:30:15.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:15.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj825 datafile 2023-12-02T01:30:15.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:15.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj826 datafile 2023-12-02T01:30:15.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:15.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj827 datafile 2023-12-02T01:30:16.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:16.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj828 datafile 2023-12-02T01:30:16.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:16.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj829 datafile 2023-12-02T01:30:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj830 datafile 2023-12-02T01:30:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj831 datafile 2023-12-02T01:30:16.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:16.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj832 datafile 2023-12-02T01:30:16.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:16.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj833 datafile 2023-12-02T01:30:16.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:16.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj834 datafile 2023-12-02T01:30:17.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:17.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj835 datafile 2023-12-02T01:30:17.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:17.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj836 datafile 2023-12-02T01:30:17.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:17.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj837 datafile 2023-12-02T01:30:17.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:17.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj838 datafile 2023-12-02T01:30:17.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:17.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj839 datafile 2023-12-02T01:30:17.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:17.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj840 datafile 2023-12-02T01:30:18.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:18.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj841 datafile 2023-12-02T01:30:18.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:18.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj842 datafile 2023-12-02T01:30:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj843 datafile 2023-12-02T01:30:18.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:18.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj844 datafile 2023-12-02T01:30:18.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:18.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj845 datafile 2023-12-02T01:30:18.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:18.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj846 datafile 2023-12-02T01:30:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:18.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj847 datafile 2023-12-02T01:30:19.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:19.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj848 datafile 2023-12-02T01:30:19.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:19.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj849 datafile 2023-12-02T01:30:19.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:19.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj850 datafile 2023-12-02T01:30:19.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:19.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj851 datafile 2023-12-02T01:30:19.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:19.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj852 datafile 2023-12-02T01:30:19.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:19.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj853 datafile 2023-12-02T01:30:19.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:19.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj854 datafile 2023-12-02T01:30:20.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:20.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj855 datafile 2023-12-02T01:30:20.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:20.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj856 datafile 2023-12-02T01:30:20.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:20.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj857 datafile 2023-12-02T01:30:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj858 datafile 2023-12-02T01:30:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj859 datafile 2023-12-02T01:30:20.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:20.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj860 datafile 2023-12-02T01:30:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj861 datafile 2023-12-02T01:30:21.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:21.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj862 datafile 2023-12-02T01:30:21.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:21.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj863 datafile 2023-12-02T01:30:21.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:21.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj864 datafile 2023-12-02T01:30:21.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:21.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj865 datafile 2023-12-02T01:30:21.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj866 datafile 2023-12-02T01:30:21.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:21.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj867 datafile 2023-12-02T01:30:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj868 datafile 2023-12-02T01:30:22.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:22.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj869 datafile 2023-12-02T01:30:22.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:22.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj870 datafile 2023-12-02T01:30:22.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:22.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj871 datafile 2023-12-02T01:30:22.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:22.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj872 datafile 2023-12-02T01:30:22.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:22.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj873 datafile 2023-12-02T01:30:23.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:23.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj874 datafile 2023-12-02T01:30:23.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:23.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj875 datafile 2023-12-02T01:30:23.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:23.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj876 datafile 2023-12-02T01:30:23.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:23.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj877 datafile 2023-12-02T01:30:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:23.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj878 datafile 2023-12-02T01:30:23.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:23.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj879 datafile 2023-12-02T01:30:23.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:23.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj880 datafile 2023-12-02T01:30:24.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:24.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj881 datafile 2023-12-02T01:30:24.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:24.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj882 datafile 2023-12-02T01:30:24.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:24.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj883 datafile 2023-12-02T01:30:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj884 datafile 2023-12-02T01:30:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj885 datafile 2023-12-02T01:30:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj886 datafile 2023-12-02T01:30:24.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:24.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj887 datafile 2023-12-02T01:30:25.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj888 datafile 2023-12-02T01:30:25.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:25.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj889 datafile 2023-12-02T01:30:25.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:25.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj890 datafile 2023-12-02T01:30:25.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:25.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj891 datafile 2023-12-02T01:30:25.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:25.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj892 datafile 2023-12-02T01:30:25.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:25.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj893 datafile 2023-12-02T01:30:26.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:26.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj894 datafile 2023-12-02T01:30:26.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:26.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj895 datafile 2023-12-02T01:30:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj896 datafile 2023-12-02T01:30:26.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:26.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj897 datafile 2023-12-02T01:30:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj898 datafile 2023-12-02T01:30:26.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:26.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj899 datafile 2023-12-02T01:30:26.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:26.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj900 datafile 2023-12-02T01:30:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj901 datafile 2023-12-02T01:30:27.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:27.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj902 datafile 2023-12-02T01:30:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj903 datafile 2023-12-02T01:30:27.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:27.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj904 datafile 2023-12-02T01:30:27.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:27.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj905 datafile 2023-12-02T01:30:27.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:27.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj906 datafile 2023-12-02T01:30:27.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:27.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj907 datafile 2023-12-02T01:30:28.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:28.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj908 datafile 2023-12-02T01:30:28.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:28.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj909 datafile 2023-12-02T01:30:28.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:28.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj910 datafile 2023-12-02T01:30:28.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:28.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj911 datafile 2023-12-02T01:30:28.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:28.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj912 datafile 2023-12-02T01:30:28.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:28.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj913 datafile 2023-12-02T01:30:29.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:29.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj914 datafile 2023-12-02T01:30:29.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj915 datafile 2023-12-02T01:30:29.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:29.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj916 datafile 2023-12-02T01:30:29.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:29.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj917 datafile 2023-12-02T01:30:29.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:29.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj918 datafile 2023-12-02T01:30:30.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:30.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj919 datafile 2023-12-02T01:30:30.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:30.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj920 datafile 2023-12-02T01:30:30.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:30.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj921 datafile 2023-12-02T01:30:30.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:30.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj922 datafile 2023-12-02T01:30:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj923 datafile 2023-12-02T01:30:30.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:30.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj924 datafile 2023-12-02T01:30:30.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:30.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj925 datafile 2023-12-02T01:30:31.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:31.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj926 datafile 2023-12-02T01:30:31.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:31.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj927 datafile 2023-12-02T01:30:31.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:31.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj928 datafile 2023-12-02T01:30:31.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:31.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj929 datafile 2023-12-02T01:30:31.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:31.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj930 datafile 2023-12-02T01:30:31.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:31.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj931 datafile 2023-12-02T01:30:31.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:31.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj932 datafile 2023-12-02T01:30:32.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:32.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj933 datafile 2023-12-02T01:30:32.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:32.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj934 datafile 2023-12-02T01:30:32.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:32.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj935 datafile 2023-12-02T01:30:32.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:32.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj936 datafile 2023-12-02T01:30:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj937 datafile 2023-12-02T01:30:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj938 datafile 2023-12-02T01:30:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj939 datafile 2023-12-02T01:30:33.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:33.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj940 datafile 2023-12-02T01:30:33.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:33.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj941 datafile 2023-12-02T01:30:33.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:33.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj942 datafile 2023-12-02T01:30:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj943 datafile 2023-12-02T01:30:33.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:33.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj944 datafile 2023-12-02T01:30:33.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:33.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj945 datafile 2023-12-02T01:30:34.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:34.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj946 datafile 2023-12-02T01:30:34.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:34.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj947 datafile 2023-12-02T01:30:34.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:34.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj948 datafile 2023-12-02T01:30:34.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:34.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj949 datafile 2023-12-02T01:30:34.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:34.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj950 datafile 2023-12-02T01:30:34.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:34.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj951 datafile 2023-12-02T01:30:35.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj952 datafile 2023-12-02T01:30:35.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:35.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj953 datafile 2023-12-02T01:30:35.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:35.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj954 datafile 2023-12-02T01:30:35.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:35.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj955 datafile 2023-12-02T01:30:35.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj956 datafile 2023-12-02T01:30:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj957 datafile 2023-12-02T01:30:35.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:35.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj958 datafile 2023-12-02T01:30:36.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:36.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj959 datafile 2023-12-02T01:30:36.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:36.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj960 datafile 2023-12-02T01:30:36.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:36.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj961 datafile 2023-12-02T01:30:36.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:36.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj962 datafile 2023-12-02T01:30:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj963 datafile 2023-12-02T01:30:36.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:36.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj964 datafile 2023-12-02T01:30:37.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:37.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj965 datafile 2023-12-02T01:30:37.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:37.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj966 datafile 2023-12-02T01:30:37.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:37.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj967 datafile 2023-12-02T01:30:37.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:37.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj968 datafile 2023-12-02T01:30:37.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:37.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj969 datafile 2023-12-02T01:30:37.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:37.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj970 datafile 2023-12-02T01:30:38.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:38.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj971 datafile 2023-12-02T01:30:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj972 datafile 2023-12-02T01:30:38.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:38.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj973 datafile 2023-12-02T01:30:38.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:38.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj974 datafile 2023-12-02T01:30:38.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:38.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj975 datafile 2023-12-02T01:30:38.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:38.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj976 datafile 2023-12-02T01:30:38.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:38.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj977 datafile 2023-12-02T01:30:39.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:39.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj978 datafile 2023-12-02T01:30:39.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:39.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj979 datafile 2023-12-02T01:30:39.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:39.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj980 datafile 2023-12-02T01:30:39.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:39.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj981 datafile 2023-12-02T01:30:39.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:39.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj982 datafile 2023-12-02T01:30:39.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:39.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj983 datafile 2023-12-02T01:30:39.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:39.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj984 datafile 2023-12-02T01:30:40.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:40.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj985 datafile 2023-12-02T01:30:40.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:40.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj986 datafile 2023-12-02T01:30:40.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:40.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj987 datafile 2023-12-02T01:30:40.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:40.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj988 datafile 2023-12-02T01:30:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj989 datafile 2023-12-02T01:30:40.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:40.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj990 datafile 2023-12-02T01:30:41.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:41.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj991 datafile 2023-12-02T01:30:41.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:41.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj992 datafile 2023-12-02T01:30:41.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:41.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj993 datafile 2023-12-02T01:30:41.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:41.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj994 datafile 2023-12-02T01:30:41.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:41.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj995 datafile 2023-12-02T01:30:41.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:41.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj996 datafile 2023-12-02T01:30:41.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:41.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj997 datafile 2023-12-02T01:30:42.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:42.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj998 datafile 2023-12-02T01:30:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj999 datafile 2023-12-02T01:30:42.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:42.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj1000 datafile 2023-12-02T01:30:42.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2023-12-02T01:30:42.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: seq 1 1000 2023-12-02T01:30:42.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:42.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj1 datafile 2023-12-02T01:30:42.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:42.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj2 datafile 2023-12-02T01:30:42.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:42.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj3 datafile 2023-12-02T01:30:43.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:43.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj4 datafile 2023-12-02T01:30:43.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:43.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj5 datafile 2023-12-02T01:30:43.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:43.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj6 datafile 2023-12-02T01:30:43.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:43.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj7 datafile 2023-12-02T01:30:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj8 datafile 2023-12-02T01:30:43.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:43.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj9 datafile 2023-12-02T01:30:43.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:43.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj10 datafile 2023-12-02T01:30:44.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:44.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj11 datafile 2023-12-02T01:30:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj12 datafile 2023-12-02T01:30:44.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:44.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj13 datafile 2023-12-02T01:30:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj14 datafile 2023-12-02T01:30:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj15 datafile 2023-12-02T01:30:44.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:44.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj16 datafile 2023-12-02T01:30:45.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:45.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj17 datafile 2023-12-02T01:30:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj18 datafile 2023-12-02T01:30:45.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:45.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj19 datafile 2023-12-02T01:30:45.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:45.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj20 datafile 2023-12-02T01:30:45.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:45.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj21 datafile 2023-12-02T01:30:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj22 datafile 2023-12-02T01:30:46.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:46.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj23 datafile 2023-12-02T01:30:46.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:46.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj24 datafile 2023-12-02T01:30:46.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:46.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj25 datafile 2023-12-02T01:30:46.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:46.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj26 datafile 2023-12-02T01:30:46.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:46.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj27 datafile 2023-12-02T01:30:46.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:46.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj28 datafile 2023-12-02T01:30:46.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:46.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj29 datafile 2023-12-02T01:30:47.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:47.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj30 datafile 2023-12-02T01:30:47.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:47.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj31 datafile 2023-12-02T01:30:47.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:47.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj32 datafile 2023-12-02T01:30:47.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:47.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj33 datafile 2023-12-02T01:30:47.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:47.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj34 datafile 2023-12-02T01:30:47.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:47.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj35 datafile 2023-12-02T01:30:47.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:47.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj36 datafile 2023-12-02T01:30:48.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:48.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj37 datafile 2023-12-02T01:30:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj38 datafile 2023-12-02T01:30:48.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:48.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj39 datafile 2023-12-02T01:30:48.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:48.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj40 datafile 2023-12-02T01:30:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj41 datafile 2023-12-02T01:30:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj42 datafile 2023-12-02T01:30:49.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:49.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj43 datafile 2023-12-02T01:30:49.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:49.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj44 datafile 2023-12-02T01:30:49.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:49.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj45 datafile 2023-12-02T01:30:49.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:49.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj46 datafile 2023-12-02T01:30:49.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:49.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj47 datafile 2023-12-02T01:30:49.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:49.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj48 datafile 2023-12-02T01:30:49.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:49.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj49 datafile 2023-12-02T01:30:50.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:50.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj50 datafile 2023-12-02T01:30:50.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:50.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj51 datafile 2023-12-02T01:30:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj52 datafile 2023-12-02T01:30:50.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:50.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj53 datafile 2023-12-02T01:30:50.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:50.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj54 datafile 2023-12-02T01:30:50.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:50.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj55 datafile 2023-12-02T01:30:50.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:50.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj56 datafile 2023-12-02T01:30:51.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:51.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj57 datafile 2023-12-02T01:30:51.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:51.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj58 datafile 2023-12-02T01:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj59 datafile 2023-12-02T01:30:51.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:51.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj60 datafile 2023-12-02T01:30:51.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:51.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj61 datafile 2023-12-02T01:30:51.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:51.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj62 datafile 2023-12-02T01:30:52.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:52.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj63 datafile 2023-12-02T01:30:52.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:52.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj64 datafile 2023-12-02T01:30:52.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:52.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj65 datafile 2023-12-02T01:30:52.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:52.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj66 datafile 2023-12-02T01:30:52.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:52.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj67 datafile 2023-12-02T01:30:52.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:52.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj68 datafile 2023-12-02T01:30:52.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:52.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj69 datafile 2023-12-02T01:30:53.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:53.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj70 datafile 2023-12-02T01:30:53.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:53.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj71 datafile 2023-12-02T01:30:53.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:53.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj72 datafile 2023-12-02T01:30:53.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:53.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj73 datafile 2023-12-02T01:30:53.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:53.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj74 datafile 2023-12-02T01:30:53.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:53.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj75 datafile 2023-12-02T01:30:53.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:53.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj76 datafile 2023-12-02T01:30:54.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:54.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj77 datafile 2023-12-02T01:30:54.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:54.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj78 datafile 2023-12-02T01:30:54.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:54.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj79 datafile 2023-12-02T01:30:54.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj80 datafile 2023-12-02T01:30:54.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:54.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj81 datafile 2023-12-02T01:30:54.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:54.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj82 datafile 2023-12-02T01:30:54.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:54.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj83 datafile 2023-12-02T01:30:55.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:55.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj84 datafile 2023-12-02T01:30:55.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:55.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj85 datafile 2023-12-02T01:30:55.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:55.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj86 datafile 2023-12-02T01:30:55.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:55.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj87 datafile 2023-12-02T01:30:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj88 datafile 2023-12-02T01:30:55.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:55.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj89 datafile 2023-12-02T01:30:56.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:56.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj90 datafile 2023-12-02T01:30:56.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:56.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj91 datafile 2023-12-02T01:30:56.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:56.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj92 datafile 2023-12-02T01:30:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj93 datafile 2023-12-02T01:30:56.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:56.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj94 datafile 2023-12-02T01:30:56.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:56.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj95 datafile 2023-12-02T01:30:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj96 datafile 2023-12-02T01:30:57.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:57.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj97 datafile 2023-12-02T01:30:57.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:57.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj98 datafile 2023-12-02T01:30:57.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:57.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj99 datafile 2023-12-02T01:30:57.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:57.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj100 datafile 2023-12-02T01:30:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj101 datafile 2023-12-02T01:30:57.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:57.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj102 datafile 2023-12-02T01:30:58.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:58.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj103 datafile 2023-12-02T01:30:58.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:58.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj104 datafile 2023-12-02T01:30:58.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:58.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj105 datafile 2023-12-02T01:30:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj106 datafile 2023-12-02T01:30:58.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:58.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj107 datafile 2023-12-02T01:30:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj108 datafile 2023-12-02T01:30:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj109 datafile 2023-12-02T01:30:59.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:59.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj110 datafile 2023-12-02T01:30:59.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:59.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj111 datafile 2023-12-02T01:30:59.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:59.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj112 datafile 2023-12-02T01:30:59.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:59.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj113 datafile 2023-12-02T01:30:59.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:59.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj114 datafile 2023-12-02T01:30:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:59.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj115 datafile 2023-12-02T01:30:59.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:30:59.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj116 datafile 2023-12-02T01:31:00.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:00.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj117 datafile 2023-12-02T01:31:00.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:00.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj118 datafile 2023-12-02T01:31:00.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:00.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj119 datafile 2023-12-02T01:31:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj120 datafile 2023-12-02T01:31:00.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:00.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj121 datafile 2023-12-02T01:31:00.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:00.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj122 datafile 2023-12-02T01:31:01.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:01.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj123 datafile 2023-12-02T01:31:01.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:01.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj124 datafile 2023-12-02T01:31:01.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:01.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj125 datafile 2023-12-02T01:31:01.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:01.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj126 datafile 2023-12-02T01:31:01.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:01.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj127 datafile 2023-12-02T01:31:01.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:01.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj128 datafile 2023-12-02T01:31:02.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:02.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj129 datafile 2023-12-02T01:31:02.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:02.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj130 datafile 2023-12-02T01:31:02.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:02.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj131 datafile 2023-12-02T01:31:02.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:02.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj132 datafile 2023-12-02T01:31:02.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:02.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj133 datafile 2023-12-02T01:31:02.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:02.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj134 datafile 2023-12-02T01:31:03.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:03.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj135 datafile 2023-12-02T01:31:03.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:03.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj136 datafile 2023-12-02T01:31:03.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:03.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj137 datafile 2023-12-02T01:31:03.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:03.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj138 datafile 2023-12-02T01:31:03.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:03.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj139 datafile 2023-12-02T01:31:03.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:03.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj140 datafile 2023-12-02T01:31:03.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:03.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj141 datafile 2023-12-02T01:31:04.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:04.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj142 datafile 2023-12-02T01:31:04.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:04.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj143 datafile 2023-12-02T01:31:04.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:04.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj144 datafile 2023-12-02T01:31:04.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:04.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj145 datafile 2023-12-02T01:31:04.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:04.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj146 datafile 2023-12-02T01:31:04.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:04.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj147 datafile 2023-12-02T01:31:05.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:05.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj148 datafile 2023-12-02T01:31:05.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:05.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj149 datafile 2023-12-02T01:31:05.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:05.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj150 datafile 2023-12-02T01:31:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj151 datafile 2023-12-02T01:31:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj152 datafile 2023-12-02T01:31:05.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:05.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj153 datafile 2023-12-02T01:31:06.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:06.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj154 datafile 2023-12-02T01:31:06.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:06.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj155 datafile 2023-12-02T01:31:06.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:06.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj156 datafile 2023-12-02T01:31:06.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:06.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj157 datafile 2023-12-02T01:31:06.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:06.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj158 datafile 2023-12-02T01:31:06.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:06.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj159 datafile 2023-12-02T01:31:06.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:06.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj160 datafile 2023-12-02T01:31:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj161 datafile 2023-12-02T01:31:07.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:07.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj162 datafile 2023-12-02T01:31:07.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:07.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj163 datafile 2023-12-02T01:31:07.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:07.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj164 datafile 2023-12-02T01:31:07.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:07.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj165 datafile 2023-12-02T01:31:07.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:07.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj166 datafile 2023-12-02T01:31:08.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:08.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj167 datafile 2023-12-02T01:31:08.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:08.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj168 datafile 2023-12-02T01:31:08.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:08.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj169 datafile 2023-12-02T01:31:08.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:08.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj170 datafile 2023-12-02T01:31:08.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:08.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj171 datafile 2023-12-02T01:31:08.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:08.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj172 datafile 2023-12-02T01:31:09.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:09.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj173 datafile 2023-12-02T01:31:09.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:09.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj174 datafile 2023-12-02T01:31:09.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:09.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj175 datafile 2023-12-02T01:31:09.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:09.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj176 datafile 2023-12-02T01:31:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj177 datafile 2023-12-02T01:31:09.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:09.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj178 datafile 2023-12-02T01:31:09.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:09.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj179 datafile 2023-12-02T01:31:10.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:10.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj180 datafile 2023-12-02T01:31:10.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:10.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj181 datafile 2023-12-02T01:31:10.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:10.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj182 datafile 2023-12-02T01:31:10.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:10.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj183 datafile 2023-12-02T01:31:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj184 datafile 2023-12-02T01:31:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj185 datafile 2023-12-02T01:31:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj186 datafile 2023-12-02T01:31:11.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:11.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj187 datafile 2023-12-02T01:31:11.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:11.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj188 datafile 2023-12-02T01:31:11.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:11.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj189 datafile 2023-12-02T01:31:11.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:11.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj190 datafile 2023-12-02T01:31:11.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:11.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj191 datafile 2023-12-02T01:31:12.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:12.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj192 datafile 2023-12-02T01:31:12.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:12.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj193 datafile 2023-12-02T01:31:12.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:12.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj194 datafile 2023-12-02T01:31:12.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:12.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj195 datafile 2023-12-02T01:31:12.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:12.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj196 datafile 2023-12-02T01:31:12.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:12.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj197 datafile 2023-12-02T01:31:12.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:12.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj198 datafile 2023-12-02T01:31:13.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:13.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj199 datafile 2023-12-02T01:31:13.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:13.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj200 datafile 2023-12-02T01:31:13.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:13.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj201 datafile 2023-12-02T01:31:13.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:13.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj202 datafile 2023-12-02T01:31:13.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:13.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj203 datafile 2023-12-02T01:31:13.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:13.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj204 datafile 2023-12-02T01:31:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj205 datafile 2023-12-02T01:31:14.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:14.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj206 datafile 2023-12-02T01:31:14.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:14.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj207 datafile 2023-12-02T01:31:14.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:14.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj208 datafile 2023-12-02T01:31:14.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:14.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj209 datafile 2023-12-02T01:31:14.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:14.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj210 datafile 2023-12-02T01:31:14.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:14.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj211 datafile 2023-12-02T01:31:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj212 datafile 2023-12-02T01:31:15.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:15.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj213 datafile 2023-12-02T01:31:15.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:15.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj214 datafile 2023-12-02T01:31:15.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:15.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj215 datafile 2023-12-02T01:31:15.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:15.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj216 datafile 2023-12-02T01:31:15.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:15.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj217 datafile 2023-12-02T01:31:15.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:15.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj218 datafile 2023-12-02T01:31:16.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:16.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj219 datafile 2023-12-02T01:31:16.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:16.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj220 datafile 2023-12-02T01:31:16.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:16.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj221 datafile 2023-12-02T01:31:16.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:16.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj222 datafile 2023-12-02T01:31:16.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:16.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj223 datafile 2023-12-02T01:31:16.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:16.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj224 datafile 2023-12-02T01:31:17.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:17.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj225 datafile 2023-12-02T01:31:17.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:17.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj226 datafile 2023-12-02T01:31:17.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:17.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj227 datafile 2023-12-02T01:31:17.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:17.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj228 datafile 2023-12-02T01:31:17.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:17.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj229 datafile 2023-12-02T01:31:17.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:17.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj230 datafile 2023-12-02T01:31:18.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:18.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj231 datafile 2023-12-02T01:31:18.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:18.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj232 datafile 2023-12-02T01:31:18.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:18.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj233 datafile 2023-12-02T01:31:18.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:18.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj234 datafile 2023-12-02T01:31:18.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:18.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj235 datafile 2023-12-02T01:31:18.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:18.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj236 datafile 2023-12-02T01:31:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj237 datafile 2023-12-02T01:31:19.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:19.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj238 datafile 2023-12-02T01:31:19.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:19.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj239 datafile 2023-12-02T01:31:19.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:19.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj240 datafile 2023-12-02T01:31:19.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:19.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj241 datafile 2023-12-02T01:31:19.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:19.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj242 datafile 2023-12-02T01:31:19.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:19.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj243 datafile 2023-12-02T01:31:20.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:20.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj244 datafile 2023-12-02T01:31:20.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:20.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj245 datafile 2023-12-02T01:31:20.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:20.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj246 datafile 2023-12-02T01:31:20.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:20.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj247 datafile 2023-12-02T01:31:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj248 datafile 2023-12-02T01:31:20.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:20.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj249 datafile 2023-12-02T01:31:20.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:20.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj250 datafile 2023-12-02T01:31:21.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:21.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj251 datafile 2023-12-02T01:31:21.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:21.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj252 datafile 2023-12-02T01:31:21.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:21.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj253 datafile 2023-12-02T01:31:21.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:21.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj254 datafile 2023-12-02T01:31:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj255 datafile 2023-12-02T01:31:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj256 datafile 2023-12-02T01:31:22.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:22.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj257 datafile 2023-12-02T01:31:22.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:22.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj258 datafile 2023-12-02T01:31:22.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:22.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj259 datafile 2023-12-02T01:31:22.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:22.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj260 datafile 2023-12-02T01:31:22.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:22.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj261 datafile 2023-12-02T01:31:22.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:22.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj262 datafile 2023-12-02T01:31:22.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:22.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj263 datafile 2023-12-02T01:31:23.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:23.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj264 datafile 2023-12-02T01:31:23.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:23.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj265 datafile 2023-12-02T01:31:23.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:23.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj266 datafile 2023-12-02T01:31:23.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:23.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj267 datafile 2023-12-02T01:31:23.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:23.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj268 datafile 2023-12-02T01:31:23.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:23.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj269 datafile 2023-12-02T01:31:23.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:23.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj270 datafile 2023-12-02T01:31:24.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:24.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj271 datafile 2023-12-02T01:31:24.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:24.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj272 datafile 2023-12-02T01:31:24.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:24.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj273 datafile 2023-12-02T01:31:24.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:24.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj274 datafile 2023-12-02T01:31:24.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:24.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj275 datafile 2023-12-02T01:31:24.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:24.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj276 datafile 2023-12-02T01:31:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj277 datafile 2023-12-02T01:31:25.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:25.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj278 datafile 2023-12-02T01:31:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj279 datafile 2023-12-02T01:31:25.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:25.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj280 datafile 2023-12-02T01:31:25.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:25.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj281 datafile 2023-12-02T01:31:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj282 datafile 2023-12-02T01:31:25.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:25.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj283 datafile 2023-12-02T01:31:26.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:26.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj284 datafile 2023-12-02T01:31:26.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:26.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj285 datafile 2023-12-02T01:31:26.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:26.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj286 datafile 2023-12-02T01:31:26.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:26.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj287 datafile 2023-12-02T01:31:26.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj288 datafile 2023-12-02T01:31:26.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:26.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj289 datafile 2023-12-02T01:31:26.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:26.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj290 datafile 2023-12-02T01:31:27.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:27.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj291 datafile 2023-12-02T01:31:27.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:27.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj292 datafile 2023-12-02T01:31:27.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:27.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj293 datafile 2023-12-02T01:31:27.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:27.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj294 datafile 2023-12-02T01:31:27.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj295 datafile 2023-12-02T01:31:27.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:27.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj296 datafile 2023-12-02T01:31:28.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:28.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj297 datafile 2023-12-02T01:31:28.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:28.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj298 datafile 2023-12-02T01:31:28.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:28.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj299 datafile 2023-12-02T01:31:28.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:28.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj300 datafile 2023-12-02T01:31:28.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:28.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj301 datafile 2023-12-02T01:31:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:28.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj302 datafile 2023-12-02T01:31:29.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:29.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj303 datafile 2023-12-02T01:31:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj304 datafile 2023-12-02T01:31:29.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:29.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj305 datafile 2023-12-02T01:31:29.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:29.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj306 datafile 2023-12-02T01:31:29.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:29.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj307 datafile 2023-12-02T01:31:29.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:29.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj308 datafile 2023-12-02T01:31:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj309 datafile 2023-12-02T01:31:30.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:30.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj310 datafile 2023-12-02T01:31:30.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:30.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj311 datafile 2023-12-02T01:31:30.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:30.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj312 datafile 2023-12-02T01:31:30.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:30.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj313 datafile 2023-12-02T01:31:30.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:30.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj314 datafile 2023-12-02T01:31:30.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:30.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj315 datafile 2023-12-02T01:31:31.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:31.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj316 datafile 2023-12-02T01:31:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj317 datafile 2023-12-02T01:31:31.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:31.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj318 datafile 2023-12-02T01:31:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj319 datafile 2023-12-02T01:31:31.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:31.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj320 datafile 2023-12-02T01:31:31.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:31.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj321 datafile 2023-12-02T01:31:31.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:31.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj322 datafile 2023-12-02T01:31:32.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:32.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj323 datafile 2023-12-02T01:31:32.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:32.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj324 datafile 2023-12-02T01:31:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj325 datafile 2023-12-02T01:31:32.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:32.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj326 datafile 2023-12-02T01:31:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj327 datafile 2023-12-02T01:31:33.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:33.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj328 datafile 2023-12-02T01:31:33.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:33.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj329 datafile 2023-12-02T01:31:33.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:33.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj330 datafile 2023-12-02T01:31:33.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:33.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj331 datafile 2023-12-02T01:31:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj332 datafile 2023-12-02T01:31:33.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:33.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj333 datafile 2023-12-02T01:31:34.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:34.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj334 datafile 2023-12-02T01:31:34.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:34.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj335 datafile 2023-12-02T01:31:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj336 datafile 2023-12-02T01:31:34.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:34.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj337 datafile 2023-12-02T01:31:34.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj338 datafile 2023-12-02T01:31:34.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:34.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj339 datafile 2023-12-02T01:31:35.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:35.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj340 datafile 2023-12-02T01:31:35.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:35.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj341 datafile 2023-12-02T01:31:35.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:35.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj342 datafile 2023-12-02T01:31:35.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:35.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj343 datafile 2023-12-02T01:31:35.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:35.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj344 datafile 2023-12-02T01:31:35.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:35.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj345 datafile 2023-12-02T01:31:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj346 datafile 2023-12-02T01:31:36.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:36.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj347 datafile 2023-12-02T01:31:36.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:36.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj348 datafile 2023-12-02T01:31:36.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:36.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj349 datafile 2023-12-02T01:31:36.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:36.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj350 datafile 2023-12-02T01:31:36.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:36.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj351 datafile 2023-12-02T01:31:36.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:36.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj352 datafile 2023-12-02T01:31:36.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:36.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj353 datafile 2023-12-02T01:31:37.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:37.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj354 datafile 2023-12-02T01:31:37.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:37.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj355 datafile 2023-12-02T01:31:37.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:37.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj356 datafile 2023-12-02T01:31:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj357 datafile 2023-12-02T01:31:37.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:37.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj358 datafile 2023-12-02T01:31:37.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:37.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj359 datafile 2023-12-02T01:31:38.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:38.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj360 datafile 2023-12-02T01:31:38.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:38.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj361 datafile 2023-12-02T01:31:38.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:38.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj362 datafile 2023-12-02T01:31:38.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:38.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj363 datafile 2023-12-02T01:31:38.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:38.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj364 datafile 2023-12-02T01:31:38.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:38.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj365 datafile 2023-12-02T01:31:39.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:39.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj366 datafile 2023-12-02T01:31:39.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:39.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj367 datafile 2023-12-02T01:31:39.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:39.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj368 datafile 2023-12-02T01:31:39.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:39.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj369 datafile 2023-12-02T01:31:39.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:39.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj370 datafile 2023-12-02T01:31:39.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:39.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj371 datafile 2023-12-02T01:31:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj372 datafile 2023-12-02T01:31:40.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:40.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj373 datafile 2023-12-02T01:31:40.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:40.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj374 datafile 2023-12-02T01:31:40.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:40.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj375 datafile 2023-12-02T01:31:40.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:40.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj376 datafile 2023-12-02T01:31:40.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:40.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj377 datafile 2023-12-02T01:31:40.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:40.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj378 datafile 2023-12-02T01:31:41.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:41.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj379 datafile 2023-12-02T01:31:41.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:41.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj380 datafile 2023-12-02T01:31:41.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:41.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj381 datafile 2023-12-02T01:31:41.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:41.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj382 datafile 2023-12-02T01:31:41.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:41.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj383 datafile 2023-12-02T01:31:41.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:41.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj384 datafile 2023-12-02T01:31:41.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:41.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj385 datafile 2023-12-02T01:31:42.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:42.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj386 datafile 2023-12-02T01:31:42.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:42.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj387 datafile 2023-12-02T01:31:42.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:42.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj388 datafile 2023-12-02T01:31:42.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:42.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj389 datafile 2023-12-02T01:31:42.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:42.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj390 datafile 2023-12-02T01:31:42.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:42.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj391 datafile 2023-12-02T01:31:43.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:43.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj392 datafile 2023-12-02T01:31:43.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:43.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj393 datafile 2023-12-02T01:31:43.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:43.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj394 datafile 2023-12-02T01:31:43.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:43.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj395 datafile 2023-12-02T01:31:43.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:43.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj396 datafile 2023-12-02T01:31:43.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:43.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj397 datafile 2023-12-02T01:31:43.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:43.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj398 datafile 2023-12-02T01:31:44.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:44.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj399 datafile 2023-12-02T01:31:44.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:44.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj400 datafile 2023-12-02T01:31:44.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:44.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj401 datafile 2023-12-02T01:31:44.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:44.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj402 datafile 2023-12-02T01:31:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj403 datafile 2023-12-02T01:31:44.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:44.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj404 datafile 2023-12-02T01:31:45.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:45.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj405 datafile 2023-12-02T01:31:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj406 datafile 2023-12-02T01:31:45.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:45.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj407 datafile 2023-12-02T01:31:45.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:45.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj408 datafile 2023-12-02T01:31:45.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:45.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj409 datafile 2023-12-02T01:31:45.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:45.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj410 datafile 2023-12-02T01:31:45.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:45.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj411 datafile 2023-12-02T01:31:46.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:46.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj412 datafile 2023-12-02T01:31:46.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:46.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj413 datafile 2023-12-02T01:31:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj414 datafile 2023-12-02T01:31:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:46.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj415 datafile 2023-12-02T01:31:46.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:46.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj416 datafile 2023-12-02T01:31:46.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:46.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj417 datafile 2023-12-02T01:31:46.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:46.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj418 datafile 2023-12-02T01:31:47.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:47.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj419 datafile 2023-12-02T01:31:47.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:47.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj420 datafile 2023-12-02T01:31:47.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:47.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj421 datafile 2023-12-02T01:31:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj422 datafile 2023-12-02T01:31:47.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:47.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj423 datafile 2023-12-02T01:31:47.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:47.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj424 datafile 2023-12-02T01:31:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj425 datafile 2023-12-02T01:31:48.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:48.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj426 datafile 2023-12-02T01:31:48.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:48.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj427 datafile 2023-12-02T01:31:48.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:48.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj428 datafile 2023-12-02T01:31:48.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:48.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj429 datafile 2023-12-02T01:31:48.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:48.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj430 datafile 2023-12-02T01:31:49.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:49.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj431 datafile 2023-12-02T01:31:49.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:49.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj432 datafile 2023-12-02T01:31:49.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:49.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj433 datafile 2023-12-02T01:31:49.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj434 datafile 2023-12-02T01:31:49.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:49.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj435 datafile 2023-12-02T01:31:49.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:49.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj436 datafile 2023-12-02T01:31:49.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:49.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj437 datafile 2023-12-02T01:31:50.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:50.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj438 datafile 2023-12-02T01:31:50.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:50.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj439 datafile 2023-12-02T01:31:50.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:50.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj440 datafile 2023-12-02T01:31:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj441 datafile 2023-12-02T01:31:50.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:50.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj442 datafile 2023-12-02T01:31:50.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:50.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj443 datafile 2023-12-02T01:31:50.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:50.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj444 datafile 2023-12-02T01:31:51.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:51.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj445 datafile 2023-12-02T01:31:51.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:51.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj446 datafile 2023-12-02T01:31:51.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:51.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj447 datafile 2023-12-02T01:31:51.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:51.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj448 datafile 2023-12-02T01:31:51.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:51.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj449 datafile 2023-12-02T01:31:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj450 datafile 2023-12-02T01:31:52.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:52.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj451 datafile 2023-12-02T01:31:52.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:52.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj452 datafile 2023-12-02T01:31:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj453 datafile 2023-12-02T01:31:52.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:52.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj454 datafile 2023-12-02T01:31:52.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:52.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj455 datafile 2023-12-02T01:31:52.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:52.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj456 datafile 2023-12-02T01:31:52.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:52.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj457 datafile 2023-12-02T01:31:53.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:53.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj458 datafile 2023-12-02T01:31:53.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:53.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj459 datafile 2023-12-02T01:31:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:53.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj460 datafile 2023-12-02T01:31:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj461 datafile 2023-12-02T01:31:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj462 datafile 2023-12-02T01:31:53.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:53.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj463 datafile 2023-12-02T01:31:53.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:53.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj464 datafile 2023-12-02T01:31:54.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:54.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj465 datafile 2023-12-02T01:31:54.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:54.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj466 datafile 2023-12-02T01:31:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj467 datafile 2023-12-02T01:31:54.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:54.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj468 datafile 2023-12-02T01:31:54.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:54.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj469 datafile 2023-12-02T01:31:54.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:54.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj470 datafile 2023-12-02T01:31:55.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:55.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj471 datafile 2023-12-02T01:31:55.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:55.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj472 datafile 2023-12-02T01:31:55.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:55.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj473 datafile 2023-12-02T01:31:55.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:55.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj474 datafile 2023-12-02T01:31:55.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:55.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj475 datafile 2023-12-02T01:31:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj476 datafile 2023-12-02T01:31:55.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:55.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj477 datafile 2023-12-02T01:31:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj478 datafile 2023-12-02T01:31:56.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:56.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj479 datafile 2023-12-02T01:31:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj480 datafile 2023-12-02T01:31:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj481 datafile 2023-12-02T01:31:56.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:56.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj482 datafile 2023-12-02T01:31:56.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:56.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj483 datafile 2023-12-02T01:31:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj484 datafile 2023-12-02T01:31:57.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj485 datafile 2023-12-02T01:31:57.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj486 datafile 2023-12-02T01:31:57.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj487 datafile 2023-12-02T01:31:57.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj488 datafile 2023-12-02T01:31:57.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj489 datafile 2023-12-02T01:31:57.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj490 datafile 2023-12-02T01:31:57.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:57.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj491 datafile 2023-12-02T01:31:58.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj492 datafile 2023-12-02T01:31:58.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:58.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj493 datafile 2023-12-02T01:31:58.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:58.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj494 datafile 2023-12-02T01:31:58.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:58.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj495 datafile 2023-12-02T01:31:58.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:58.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj496 datafile 2023-12-02T01:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj497 datafile 2023-12-02T01:31:59.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:59.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj498 datafile 2023-12-02T01:31:59.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:59.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj499 datafile 2023-12-02T01:31:59.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:59.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj500 datafile 2023-12-02T01:31:59.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:59.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj501 datafile 2023-12-02T01:31:59.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:59.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj502 datafile 2023-12-02T01:31:59.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:31:59.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj503 datafile 2023-12-02T01:32:00.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:00.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj504 datafile 2023-12-02T01:32:00.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:00.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj505 datafile 2023-12-02T01:32:00.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:00.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj506 datafile 2023-12-02T01:32:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj507 datafile 2023-12-02T01:32:00.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:00.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj508 datafile 2023-12-02T01:32:00.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:00.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj509 datafile 2023-12-02T01:32:00.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:00.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj510 datafile 2023-12-02T01:32:01.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:01.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj511 datafile 2023-12-02T01:32:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj512 datafile 2023-12-02T01:32:01.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:01.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj513 datafile 2023-12-02T01:32:01.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:01.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj514 datafile 2023-12-02T01:32:01.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:01.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj515 datafile 2023-12-02T01:32:01.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:01.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj516 datafile 2023-12-02T01:32:01.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:01.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj517 datafile 2023-12-02T01:32:02.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:02.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj518 datafile 2023-12-02T01:32:02.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:02.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj519 datafile 2023-12-02T01:32:02.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:02.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj520 datafile 2023-12-02T01:32:02.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:02.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj521 datafile 2023-12-02T01:32:02.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:02.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj522 datafile 2023-12-02T01:32:02.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:02.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj523 datafile 2023-12-02T01:32:02.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:02.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj524 datafile 2023-12-02T01:32:03.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:03.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj525 datafile 2023-12-02T01:32:03.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:03.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj526 datafile 2023-12-02T01:32:03.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:03.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj527 datafile 2023-12-02T01:32:03.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:03.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj528 datafile 2023-12-02T01:32:03.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:03.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj529 datafile 2023-12-02T01:32:04.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:04.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj530 datafile 2023-12-02T01:32:04.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:04.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj531 datafile 2023-12-02T01:32:04.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:04.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj532 datafile 2023-12-02T01:32:04.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:04.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj533 datafile 2023-12-02T01:32:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj534 datafile 2023-12-02T01:32:04.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:04.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj535 datafile 2023-12-02T01:32:05.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:05.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj536 datafile 2023-12-02T01:32:05.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:05.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj537 datafile 2023-12-02T01:32:05.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:05.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj538 datafile 2023-12-02T01:32:05.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:05.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj539 datafile 2023-12-02T01:32:05.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:05.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj540 datafile 2023-12-02T01:32:05.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:05.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj541 datafile 2023-12-02T01:32:05.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:05.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj542 datafile 2023-12-02T01:32:06.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:06.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj543 datafile 2023-12-02T01:32:06.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:06.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj544 datafile 2023-12-02T01:32:06.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:06.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj545 datafile 2023-12-02T01:32:06.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:06.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj546 datafile 2023-12-02T01:32:06.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:06.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj547 datafile 2023-12-02T01:32:06.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:06.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj548 datafile 2023-12-02T01:32:06.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:06.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj549 datafile 2023-12-02T01:32:07.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:07.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj550 datafile 2023-12-02T01:32:07.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:07.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj551 datafile 2023-12-02T01:32:07.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:07.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj552 datafile 2023-12-02T01:32:07.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:07.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj553 datafile 2023-12-02T01:32:07.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:07.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj554 datafile 2023-12-02T01:32:07.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:07.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj555 datafile 2023-12-02T01:32:08.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:08.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj556 datafile 2023-12-02T01:32:08.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:08.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj557 datafile 2023-12-02T01:32:08.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:08.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj558 datafile 2023-12-02T01:32:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj559 datafile 2023-12-02T01:32:08.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:08.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj560 datafile 2023-12-02T01:32:08.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:08.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj561 datafile 2023-12-02T01:32:08.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:08.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj562 datafile 2023-12-02T01:32:09.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:09.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj563 datafile 2023-12-02T01:32:09.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:09.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj564 datafile 2023-12-02T01:32:09.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:09.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj565 datafile 2023-12-02T01:32:09.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:09.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj566 datafile 2023-12-02T01:32:09.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:09.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj567 datafile 2023-12-02T01:32:09.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:09.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj568 datafile 2023-12-02T01:32:09.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:09.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj569 datafile 2023-12-02T01:32:10.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:10.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj570 datafile 2023-12-02T01:32:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj571 datafile 2023-12-02T01:32:10.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:10.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj572 datafile 2023-12-02T01:32:10.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:10.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj573 datafile 2023-12-02T01:32:10.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:10.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj574 datafile 2023-12-02T01:32:10.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:10.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj575 datafile 2023-12-02T01:32:11.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:11.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj576 datafile 2023-12-02T01:32:11.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:11.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj577 datafile 2023-12-02T01:32:11.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:11.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj578 datafile 2023-12-02T01:32:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj579 datafile 2023-12-02T01:32:11.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:11.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj580 datafile 2023-12-02T01:32:11.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:11.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj581 datafile 2023-12-02T01:32:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj582 datafile 2023-12-02T01:32:12.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:12.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj583 datafile 2023-12-02T01:32:12.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:12.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj584 datafile 2023-12-02T01:32:12.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:12.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj585 datafile 2023-12-02T01:32:12.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:12.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj586 datafile 2023-12-02T01:32:12.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:12.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj587 datafile 2023-12-02T01:32:12.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:12.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj588 datafile 2023-12-02T01:32:12.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:12.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj589 datafile 2023-12-02T01:32:13.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:13.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj590 datafile 2023-12-02T01:32:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj591 datafile 2023-12-02T01:32:13.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:13.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj592 datafile 2023-12-02T01:32:13.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:13.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj593 datafile 2023-12-02T01:32:13.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:13.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj594 datafile 2023-12-02T01:32:13.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:13.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj595 datafile 2023-12-02T01:32:14.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:14.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj596 datafile 2023-12-02T01:32:14.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:14.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj597 datafile 2023-12-02T01:32:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj598 datafile 2023-12-02T01:32:14.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:14.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj599 datafile 2023-12-02T01:32:14.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:14.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj600 datafile 2023-12-02T01:32:14.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:14.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj601 datafile 2023-12-02T01:32:15.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:15.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj602 datafile 2023-12-02T01:32:15.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:15.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj603 datafile 2023-12-02T01:32:15.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:15.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj604 datafile 2023-12-02T01:32:15.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:15.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj605 datafile 2023-12-02T01:32:15.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:15.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj606 datafile 2023-12-02T01:32:15.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:15.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj607 datafile 2023-12-02T01:32:15.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:15.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj608 datafile 2023-12-02T01:32:16.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:16.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj609 datafile 2023-12-02T01:32:16.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:16.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj610 datafile 2023-12-02T01:32:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj611 datafile 2023-12-02T01:32:16.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:16.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj612 datafile 2023-12-02T01:32:16.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:16.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj613 datafile 2023-12-02T01:32:16.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:16.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj614 datafile 2023-12-02T01:32:16.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:16.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj615 datafile 2023-12-02T01:32:17.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:17.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj616 datafile 2023-12-02T01:32:17.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:17.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj617 datafile 2023-12-02T01:32:17.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:17.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj618 datafile 2023-12-02T01:32:17.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:17.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj619 datafile 2023-12-02T01:32:17.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:17.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj620 datafile 2023-12-02T01:32:17.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:17.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj621 datafile 2023-12-02T01:32:17.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:17.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj622 datafile 2023-12-02T01:32:18.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:18.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj623 datafile 2023-12-02T01:32:18.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:18.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj624 datafile 2023-12-02T01:32:18.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:18.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj625 datafile 2023-12-02T01:32:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj626 datafile 2023-12-02T01:32:18.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:18.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj627 datafile 2023-12-02T01:32:18.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:18.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj628 datafile 2023-12-02T01:32:19.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:19.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj629 datafile 2023-12-02T01:32:19.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:19.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj630 datafile 2023-12-02T01:32:19.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:19.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj631 datafile 2023-12-02T01:32:19.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:19.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj632 datafile 2023-12-02T01:32:19.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:19.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj633 datafile 2023-12-02T01:32:19.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:19.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj634 datafile 2023-12-02T01:32:19.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:19.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj635 datafile 2023-12-02T01:32:20.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:20.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj636 datafile 2023-12-02T01:32:20.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:20.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj637 datafile 2023-12-02T01:32:20.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:20.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj638 datafile 2023-12-02T01:32:20.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:20.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj639 datafile 2023-12-02T01:32:20.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:20.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj640 datafile 2023-12-02T01:32:20.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:20.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj641 datafile 2023-12-02T01:32:21.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:21.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj642 datafile 2023-12-02T01:32:21.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:21.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj643 datafile 2023-12-02T01:32:21.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:21.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj644 datafile 2023-12-02T01:32:21.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:21.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj645 datafile 2023-12-02T01:32:21.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:21.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj646 datafile 2023-12-02T01:32:21.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:21.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj647 datafile 2023-12-02T01:32:21.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:21.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj648 datafile 2023-12-02T01:32:22.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:22.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj649 datafile 2023-12-02T01:32:22.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:22.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj650 datafile 2023-12-02T01:32:22.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:22.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj651 datafile 2023-12-02T01:32:22.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:22.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj652 datafile 2023-12-02T01:32:22.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:22.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj653 datafile 2023-12-02T01:32:22.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:22.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj654 datafile 2023-12-02T01:32:22.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:22.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj655 datafile 2023-12-02T01:32:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj656 datafile 2023-12-02T01:32:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj657 datafile 2023-12-02T01:32:23.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:23.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj658 datafile 2023-12-02T01:32:23.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:23.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj659 datafile 2023-12-02T01:32:23.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:23.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj660 datafile 2023-12-02T01:32:23.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:23.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj661 datafile 2023-12-02T01:32:23.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:23.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj662 datafile 2023-12-02T01:32:24.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:24.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj663 datafile 2023-12-02T01:32:24.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:24.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj664 datafile 2023-12-02T01:32:24.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:24.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj665 datafile 2023-12-02T01:32:24.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:24.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj666 datafile 2023-12-02T01:32:24.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:24.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj667 datafile 2023-12-02T01:32:24.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:24.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj668 datafile 2023-12-02T01:32:25.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:25.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj669 datafile 2023-12-02T01:32:25.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:25.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj670 datafile 2023-12-02T01:32:25.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:25.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj671 datafile 2023-12-02T01:32:25.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:25.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj672 datafile 2023-12-02T01:32:25.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:25.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj673 datafile 2023-12-02T01:32:25.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:25.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj674 datafile 2023-12-02T01:32:25.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:25.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj675 datafile 2023-12-02T01:32:26.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:26.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj676 datafile 2023-12-02T01:32:26.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:26.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj677 datafile 2023-12-02T01:32:26.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:26.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj678 datafile 2023-12-02T01:32:26.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:26.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj679 datafile 2023-12-02T01:32:26.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:26.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj680 datafile 2023-12-02T01:32:26.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:26.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj681 datafile 2023-12-02T01:32:27.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:27.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj682 datafile 2023-12-02T01:32:27.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:27.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj683 datafile 2023-12-02T01:32:27.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:27.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj684 datafile 2023-12-02T01:32:27.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:27.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj685 datafile 2023-12-02T01:32:27.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:27.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj686 datafile 2023-12-02T01:32:27.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:27.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj687 datafile 2023-12-02T01:32:27.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:27.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj688 datafile 2023-12-02T01:32:28.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:28.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj689 datafile 2023-12-02T01:32:28.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:28.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj690 datafile 2023-12-02T01:32:28.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:28.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj691 datafile 2023-12-02T01:32:28.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:28.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj692 datafile 2023-12-02T01:32:28.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:28.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj693 datafile 2023-12-02T01:32:28.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:28.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj694 datafile 2023-12-02T01:32:28.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:28.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj695 datafile 2023-12-02T01:32:29.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:29.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj696 datafile 2023-12-02T01:32:29.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:29.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj697 datafile 2023-12-02T01:32:29.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:29.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj698 datafile 2023-12-02T01:32:29.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:29.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj699 datafile 2023-12-02T01:32:29.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:29.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj700 datafile 2023-12-02T01:32:29.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:29.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj701 datafile 2023-12-02T01:32:29.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:29.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj702 datafile 2023-12-02T01:32:30.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:30.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj703 datafile 2023-12-02T01:32:30.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:30.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj704 datafile 2023-12-02T01:32:30.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:30.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj705 datafile 2023-12-02T01:32:30.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:30.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj706 datafile 2023-12-02T01:32:30.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:30.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj707 datafile 2023-12-02T01:32:30.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:30.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj708 datafile 2023-12-02T01:32:30.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:30.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj709 datafile 2023-12-02T01:32:31.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:31.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj710 datafile 2023-12-02T01:32:31.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:31.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj711 datafile 2023-12-02T01:32:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj712 datafile 2023-12-02T01:32:31.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:31.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj713 datafile 2023-12-02T01:32:31.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:31.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj714 datafile 2023-12-02T01:32:31.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:31.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj715 datafile 2023-12-02T01:32:32.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:32.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj716 datafile 2023-12-02T01:32:32.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:32.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj717 datafile 2023-12-02T01:32:32.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:32.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj718 datafile 2023-12-02T01:32:32.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:32.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj719 datafile 2023-12-02T01:32:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj720 datafile 2023-12-02T01:32:32.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:32.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj721 datafile 2023-12-02T01:32:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj722 datafile 2023-12-02T01:32:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj723 datafile 2023-12-02T01:32:33.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:33.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj724 datafile 2023-12-02T01:32:33.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:33.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj725 datafile 2023-12-02T01:32:33.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:33.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj726 datafile 2023-12-02T01:32:33.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:33.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj727 datafile 2023-12-02T01:32:33.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:33.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj728 datafile 2023-12-02T01:32:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj729 datafile 2023-12-02T01:32:34.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:34.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj730 datafile 2023-12-02T01:32:34.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:34.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj731 datafile 2023-12-02T01:32:34.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:34.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj732 datafile 2023-12-02T01:32:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj733 datafile 2023-12-02T01:32:35.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:35.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj734 datafile 2023-12-02T01:32:35.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:35.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj735 datafile 2023-12-02T01:32:35.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:35.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj736 datafile 2023-12-02T01:32:35.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:35.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj737 datafile 2023-12-02T01:32:35.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:35.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj738 datafile 2023-12-02T01:32:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj739 datafile 2023-12-02T01:32:36.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:36.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj740 datafile 2023-12-02T01:32:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj741 datafile 2023-12-02T01:32:36.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:36.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj742 datafile 2023-12-02T01:32:36.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:36.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj743 datafile 2023-12-02T01:32:36.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:36.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj744 datafile 2023-12-02T01:32:36.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:36.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj745 datafile 2023-12-02T01:32:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj746 datafile 2023-12-02T01:32:37.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:37.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj747 datafile 2023-12-02T01:32:37.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:37.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj748 datafile 2023-12-02T01:32:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj749 datafile 2023-12-02T01:32:37.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:37.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj750 datafile 2023-12-02T01:32:37.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:37.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj751 datafile 2023-12-02T01:32:37.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:37.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj752 datafile 2023-12-02T01:32:38.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:38.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj753 datafile 2023-12-02T01:32:38.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:38.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj754 datafile 2023-12-02T01:32:38.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:38.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj755 datafile 2023-12-02T01:32:38.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:38.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj756 datafile 2023-12-02T01:32:38.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:38.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj757 datafile 2023-12-02T01:32:38.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:38.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj758 datafile 2023-12-02T01:32:38.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:38.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj759 datafile 2023-12-02T01:32:39.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:39.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj760 datafile 2023-12-02T01:32:39.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:39.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj761 datafile 2023-12-02T01:32:39.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:39.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj762 datafile 2023-12-02T01:32:39.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:39.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj763 datafile 2023-12-02T01:32:39.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:39.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj764 datafile 2023-12-02T01:32:39.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:39.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj765 datafile 2023-12-02T01:32:39.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:39.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj766 datafile 2023-12-02T01:32:40.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:40.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj767 datafile 2023-12-02T01:32:40.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:40.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj768 datafile 2023-12-02T01:32:40.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:40.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj769 datafile 2023-12-02T01:32:40.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:40.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj770 datafile 2023-12-02T01:32:40.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:40.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj771 datafile 2023-12-02T01:32:40.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:40.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj772 datafile 2023-12-02T01:32:41.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:41.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj773 datafile 2023-12-02T01:32:41.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:41.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj774 datafile 2023-12-02T01:32:41.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:41.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj775 datafile 2023-12-02T01:32:41.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:41.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj776 datafile 2023-12-02T01:32:41.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:41.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj777 datafile 2023-12-02T01:32:41.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:41.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj778 datafile 2023-12-02T01:32:41.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:41.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj779 datafile 2023-12-02T01:32:42.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:42.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj780 datafile 2023-12-02T01:32:42.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:42.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj781 datafile 2023-12-02T01:32:42.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:42.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj782 datafile 2023-12-02T01:32:42.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:42.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj783 datafile 2023-12-02T01:32:42.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:42.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj784 datafile 2023-12-02T01:32:42.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:42.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj785 datafile 2023-12-02T01:32:42.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:42.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj786 datafile 2023-12-02T01:32:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj787 datafile 2023-12-02T01:32:43.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:43.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj788 datafile 2023-12-02T01:32:43.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:43.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj789 datafile 2023-12-02T01:32:43.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:43.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj790 datafile 2023-12-02T01:32:43.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:43.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj791 datafile 2023-12-02T01:32:43.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:43.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj792 datafile 2023-12-02T01:32:44.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:44.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj793 datafile 2023-12-02T01:32:44.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:44.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj794 datafile 2023-12-02T01:32:44.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:44.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj795 datafile 2023-12-02T01:32:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:44.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj796 datafile 2023-12-02T01:32:44.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:44.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj797 datafile 2023-12-02T01:32:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:44.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj798 datafile 2023-12-02T01:32:44.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:44.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj799 datafile 2023-12-02T01:32:45.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:45.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj800 datafile 2023-12-02T01:32:45.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:45.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj801 datafile 2023-12-02T01:32:45.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:45.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj802 datafile 2023-12-02T01:32:45.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:45.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj803 datafile 2023-12-02T01:32:45.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:45.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj804 datafile 2023-12-02T01:32:45.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:45.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj805 datafile 2023-12-02T01:32:45.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:45.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj806 datafile 2023-12-02T01:32:46.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:46.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj807 datafile 2023-12-02T01:32:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj808 datafile 2023-12-02T01:32:46.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:46.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj809 datafile 2023-12-02T01:32:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj810 datafile 2023-12-02T01:32:46.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:46.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj811 datafile 2023-12-02T01:32:46.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:46.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj812 datafile 2023-12-02T01:32:47.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:47.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj813 datafile 2023-12-02T01:32:47.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:47.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj814 datafile 2023-12-02T01:32:47.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:47.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj815 datafile 2023-12-02T01:32:47.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:47.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj816 datafile 2023-12-02T01:32:47.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:47.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj817 datafile 2023-12-02T01:32:47.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:47.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj818 datafile 2023-12-02T01:32:47.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:47.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj819 datafile 2023-12-02T01:32:48.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj820 datafile 2023-12-02T01:32:48.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:48.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj821 datafile 2023-12-02T01:32:48.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:48.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj822 datafile 2023-12-02T01:32:48.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:48.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj823 datafile 2023-12-02T01:32:48.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:48.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj824 datafile 2023-12-02T01:32:48.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:48.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj825 datafile 2023-12-02T01:32:48.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:48.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj826 datafile 2023-12-02T01:32:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj827 datafile 2023-12-02T01:32:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj828 datafile 2023-12-02T01:32:49.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:49.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj829 datafile 2023-12-02T01:32:49.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:49.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj830 datafile 2023-12-02T01:32:49.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:49.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj831 datafile 2023-12-02T01:32:49.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:49.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj832 datafile 2023-12-02T01:32:49.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:49.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj833 datafile 2023-12-02T01:32:50.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:50.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj834 datafile 2023-12-02T01:32:50.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:50.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj835 datafile 2023-12-02T01:32:50.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:50.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj836 datafile 2023-12-02T01:32:50.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:50.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj837 datafile 2023-12-02T01:32:50.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:50.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj838 datafile 2023-12-02T01:32:50.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:50.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj839 datafile 2023-12-02T01:32:51.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj840 datafile 2023-12-02T01:32:51.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:51.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj841 datafile 2023-12-02T01:32:51.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:51.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj842 datafile 2023-12-02T01:32:51.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:51.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj843 datafile 2023-12-02T01:32:51.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:51.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj844 datafile 2023-12-02T01:32:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj845 datafile 2023-12-02T01:32:51.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:51.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj846 datafile 2023-12-02T01:32:52.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:52.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj847 datafile 2023-12-02T01:32:52.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:52.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj848 datafile 2023-12-02T01:32:52.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:52.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj849 datafile 2023-12-02T01:32:52.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:52.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj850 datafile 2023-12-02T01:32:52.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:52.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj851 datafile 2023-12-02T01:32:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj852 datafile 2023-12-02T01:32:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj853 datafile 2023-12-02T01:32:53.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:53.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj854 datafile 2023-12-02T01:32:53.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:53.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj855 datafile 2023-12-02T01:32:53.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:53.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj856 datafile 2023-12-02T01:32:53.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:53.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj857 datafile 2023-12-02T01:32:53.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:53.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj858 datafile 2023-12-02T01:32:53.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:53.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj859 datafile 2023-12-02T01:32:54.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:54.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj860 datafile 2023-12-02T01:32:54.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:54.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj861 datafile 2023-12-02T01:32:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj862 datafile 2023-12-02T01:32:54.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:54.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj863 datafile 2023-12-02T01:32:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj864 datafile 2023-12-02T01:32:54.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:54.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj865 datafile 2023-12-02T01:32:54.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:54.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj866 datafile 2023-12-02T01:32:55.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:55.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj867 datafile 2023-12-02T01:32:55.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:55.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj868 datafile 2023-12-02T01:32:55.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:55.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj869 datafile 2023-12-02T01:32:55.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:55.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj870 datafile 2023-12-02T01:32:55.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:55.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj871 datafile 2023-12-02T01:32:55.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:55.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj872 datafile 2023-12-02T01:32:55.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:55.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj873 datafile 2023-12-02T01:32:56.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:56.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj874 datafile 2023-12-02T01:32:56.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj875 datafile 2023-12-02T01:32:56.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:56.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj876 datafile 2023-12-02T01:32:56.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:56.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj877 datafile 2023-12-02T01:32:56.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:56.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj878 datafile 2023-12-02T01:32:56.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:56.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj879 datafile 2023-12-02T01:32:57.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:57.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj880 datafile 2023-12-02T01:32:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj881 datafile 2023-12-02T01:32:57.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:57.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj882 datafile 2023-12-02T01:32:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj883 datafile 2023-12-02T01:32:57.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:57.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj884 datafile 2023-12-02T01:32:57.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:57.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj885 datafile 2023-12-02T01:32:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj886 datafile 2023-12-02T01:32:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj887 datafile 2023-12-02T01:32:58.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:58.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj888 datafile 2023-12-02T01:32:58.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:58.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj889 datafile 2023-12-02T01:32:58.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:58.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj890 datafile 2023-12-02T01:32:58.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:58.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj891 datafile 2023-12-02T01:32:58.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:58.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj892 datafile 2023-12-02T01:32:58.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:58.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj893 datafile 2023-12-02T01:32:59.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:59.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj894 datafile 2023-12-02T01:32:59.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:59.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj895 datafile 2023-12-02T01:32:59.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:59.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj896 datafile 2023-12-02T01:32:59.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:59.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj897 datafile 2023-12-02T01:32:59.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:59.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj898 datafile 2023-12-02T01:32:59.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:32:59.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj899 datafile 2023-12-02T01:33:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj900 datafile 2023-12-02T01:33:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj901 datafile 2023-12-02T01:33:00.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:00.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj902 datafile 2023-12-02T01:33:00.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:00.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj903 datafile 2023-12-02T01:33:00.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:00.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj904 datafile 2023-12-02T01:33:00.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:00.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj905 datafile 2023-12-02T01:33:00.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:00.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj906 datafile 2023-12-02T01:33:01.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:01.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj907 datafile 2023-12-02T01:33:01.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:01.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj908 datafile 2023-12-02T01:33:01.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:01.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj909 datafile 2023-12-02T01:33:01.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:01.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj910 datafile 2023-12-02T01:33:01.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:01.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj911 datafile 2023-12-02T01:33:01.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:01.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj912 datafile 2023-12-02T01:33:01.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:01.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj913 datafile 2023-12-02T01:33:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj914 datafile 2023-12-02T01:33:02.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:02.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj915 datafile 2023-12-02T01:33:02.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:02.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj916 datafile 2023-12-02T01:33:02.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:02.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj917 datafile 2023-12-02T01:33:02.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:02.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj918 datafile 2023-12-02T01:33:02.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:02.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj919 datafile 2023-12-02T01:33:03.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:03.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj920 datafile 2023-12-02T01:33:03.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:03.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj921 datafile 2023-12-02T01:33:03.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:03.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj922 datafile 2023-12-02T01:33:03.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:03.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj923 datafile 2023-12-02T01:33:03.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:03.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj924 datafile 2023-12-02T01:33:03.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:03.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj925 datafile 2023-12-02T01:33:03.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:03.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj926 datafile 2023-12-02T01:33:04.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:04.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj927 datafile 2023-12-02T01:33:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj928 datafile 2023-12-02T01:33:04.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:04.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj929 datafile 2023-12-02T01:33:04.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:04.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj930 datafile 2023-12-02T01:33:04.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:04.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj931 datafile 2023-12-02T01:33:04.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:04.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj932 datafile 2023-12-02T01:33:05.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:05.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj933 datafile 2023-12-02T01:33:05.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:05.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj934 datafile 2023-12-02T01:33:05.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:05.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj935 datafile 2023-12-02T01:33:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj936 datafile 2023-12-02T01:33:05.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:05.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj937 datafile 2023-12-02T01:33:05.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:05.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj938 datafile 2023-12-02T01:33:06.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:06.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj939 datafile 2023-12-02T01:33:06.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:06.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj940 datafile 2023-12-02T01:33:06.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:06.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj941 datafile 2023-12-02T01:33:06.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:06.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj942 datafile 2023-12-02T01:33:06.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:06.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj943 datafile 2023-12-02T01:33:06.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:06.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj944 datafile 2023-12-02T01:33:07.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:07.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj945 datafile 2023-12-02T01:33:07.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:07.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj946 datafile 2023-12-02T01:33:07.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:07.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj947 datafile 2023-12-02T01:33:07.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:07.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj948 datafile 2023-12-02T01:33:07.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:07.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj949 datafile 2023-12-02T01:33:07.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:07.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj950 datafile 2023-12-02T01:33:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj951 datafile 2023-12-02T01:33:08.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:08.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj952 datafile 2023-12-02T01:33:08.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:08.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj953 datafile 2023-12-02T01:33:08.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:08.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj954 datafile 2023-12-02T01:33:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj955 datafile 2023-12-02T01:33:08.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:08.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj956 datafile 2023-12-02T01:33:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj957 datafile 2023-12-02T01:33:09.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:09.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj958 datafile 2023-12-02T01:33:09.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:09.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj959 datafile 2023-12-02T01:33:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj960 datafile 2023-12-02T01:33:09.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:09.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj961 datafile 2023-12-02T01:33:09.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:09.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj962 datafile 2023-12-02T01:33:09.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:09.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj963 datafile 2023-12-02T01:33:09.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:09.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj964 datafile 2023-12-02T01:33:10.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj965 datafile 2023-12-02T01:33:10.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:10.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj966 datafile 2023-12-02T01:33:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj967 datafile 2023-12-02T01:33:10.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:10.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj968 datafile 2023-12-02T01:33:10.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:10.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj969 datafile 2023-12-02T01:33:10.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:10.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj970 datafile 2023-12-02T01:33:11.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:11.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj971 datafile 2023-12-02T01:33:11.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:11.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj972 datafile 2023-12-02T01:33:11.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:11.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj973 datafile 2023-12-02T01:33:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj974 datafile 2023-12-02T01:33:11.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:11.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj975 datafile 2023-12-02T01:33:11.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:11.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj976 datafile 2023-12-02T01:33:11.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:11.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj977 datafile 2023-12-02T01:33:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj978 datafile 2023-12-02T01:33:12.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:12.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj979 datafile 2023-12-02T01:33:12.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:12.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj980 datafile 2023-12-02T01:33:12.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:12.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj981 datafile 2023-12-02T01:33:12.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:12.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj982 datafile 2023-12-02T01:33:12.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:12.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj983 datafile 2023-12-02T01:33:12.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:12.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj984 datafile 2023-12-02T01:33:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj985 datafile 2023-12-02T01:33:13.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:13.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj986 datafile 2023-12-02T01:33:13.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:13.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj987 datafile 2023-12-02T01:33:13.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:13.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj988 datafile 2023-12-02T01:33:13.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:13.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj989 datafile 2023-12-02T01:33:13.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:13.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj990 datafile 2023-12-02T01:33:13.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:13.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj991 datafile 2023-12-02T01:33:14.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:14.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj992 datafile 2023-12-02T01:33:14.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:14.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj993 datafile 2023-12-02T01:33:14.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:14.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj994 datafile 2023-12-02T01:33:14.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:14.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj995 datafile 2023-12-02T01:33:14.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:14.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj996 datafile 2023-12-02T01:33:14.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:14.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj997 datafile 2023-12-02T01:33:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj998 datafile 2023-12-02T01:33:15.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:15.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj999 datafile 2023-12-02T01:33:15.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:15.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj1000 datafile 2023-12-02T01:33:15.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2023-12-02T01:33:15.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: seq 1 1000 2023-12-02T01:33:15.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:15.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj1 datafile 2023-12-02T01:33:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj2 datafile 2023-12-02T01:33:15.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:15.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj3 datafile 2023-12-02T01:33:15.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:15.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj4 datafile 2023-12-02T01:33:16.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:16.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj5 datafile 2023-12-02T01:33:16.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:16.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj6 datafile 2023-12-02T01:33:16.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:16.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj7 datafile 2023-12-02T01:33:16.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:16.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj8 datafile 2023-12-02T01:33:16.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:16.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj9 datafile 2023-12-02T01:33:16.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:16.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj10 datafile 2023-12-02T01:33:16.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:16.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj11 datafile 2023-12-02T01:33:17.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:17.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj12 datafile 2023-12-02T01:33:17.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:17.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj13 datafile 2023-12-02T01:33:17.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:17.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj14 datafile 2023-12-02T01:33:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj15 datafile 2023-12-02T01:33:17.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:17.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj16 datafile 2023-12-02T01:33:17.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:17.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj17 datafile 2023-12-02T01:33:18.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:18.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj18 datafile 2023-12-02T01:33:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj19 datafile 2023-12-02T01:33:18.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:18.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj20 datafile 2023-12-02T01:33:18.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:18.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj21 datafile 2023-12-02T01:33:18.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:18.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj22 datafile 2023-12-02T01:33:18.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:18.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj23 datafile 2023-12-02T01:33:18.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:18.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj24 datafile 2023-12-02T01:33:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj25 datafile 2023-12-02T01:33:19.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:19.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj26 datafile 2023-12-02T01:33:19.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:19.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj27 datafile 2023-12-02T01:33:19.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:19.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj28 datafile 2023-12-02T01:33:19.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:19.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj29 datafile 2023-12-02T01:33:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj30 datafile 2023-12-02T01:33:19.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:19.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj31 datafile 2023-12-02T01:33:20.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:20.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj32 datafile 2023-12-02T01:33:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj33 datafile 2023-12-02T01:33:20.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:20.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj34 datafile 2023-12-02T01:33:20.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:20.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj35 datafile 2023-12-02T01:33:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj36 datafile 2023-12-02T01:33:20.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:20.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj37 datafile 2023-12-02T01:33:21.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:21.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj38 datafile 2023-12-02T01:33:21.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:21.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj39 datafile 2023-12-02T01:33:21.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:21.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj40 datafile 2023-12-02T01:33:21.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:21.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj41 datafile 2023-12-02T01:33:21.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:21.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj42 datafile 2023-12-02T01:33:21.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:21.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj43 datafile 2023-12-02T01:33:21.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:21.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj44 datafile 2023-12-02T01:33:22.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:22.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj45 datafile 2023-12-02T01:33:22.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj46 datafile 2023-12-02T01:33:22.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:22.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj47 datafile 2023-12-02T01:33:22.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:22.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj48 datafile 2023-12-02T01:33:22.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj49 datafile 2023-12-02T01:33:22.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:22.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj50 datafile 2023-12-02T01:33:22.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:22.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj51 datafile 2023-12-02T01:33:23.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:23.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj52 datafile 2023-12-02T01:33:23.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:23.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj53 datafile 2023-12-02T01:33:23.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:23.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj54 datafile 2023-12-02T01:33:23.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:23.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj55 datafile 2023-12-02T01:33:23.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:23.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj56 datafile 2023-12-02T01:33:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj57 datafile 2023-12-02T01:33:23.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:23.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj58 datafile 2023-12-02T01:33:24.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:24.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj59 datafile 2023-12-02T01:33:24.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:24.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj60 datafile 2023-12-02T01:33:24.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:24.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj61 datafile 2023-12-02T01:33:24.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:24.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj62 datafile 2023-12-02T01:33:24.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:24.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj63 datafile 2023-12-02T01:33:24.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:24.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj64 datafile 2023-12-02T01:33:25.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:25.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj65 datafile 2023-12-02T01:33:25.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:25.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj66 datafile 2023-12-02T01:33:25.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:25.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj67 datafile 2023-12-02T01:33:25.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:25.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj68 datafile 2023-12-02T01:33:25.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:25.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj69 datafile 2023-12-02T01:33:25.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:25.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj70 datafile 2023-12-02T01:33:25.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:25.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj71 datafile 2023-12-02T01:33:26.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:26.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj72 datafile 2023-12-02T01:33:26.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:26.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj73 datafile 2023-12-02T01:33:26.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:26.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj74 datafile 2023-12-02T01:33:26.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:26.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj75 datafile 2023-12-02T01:33:26.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:26.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj76 datafile 2023-12-02T01:33:26.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:26.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj77 datafile 2023-12-02T01:33:27.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:27.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj78 datafile 2023-12-02T01:33:27.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:27.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj79 datafile 2023-12-02T01:33:27.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:27.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj80 datafile 2023-12-02T01:33:27.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:27.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj81 datafile 2023-12-02T01:33:27.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:27.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj82 datafile 2023-12-02T01:33:27.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:27.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj83 datafile 2023-12-02T01:33:27.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:27.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj84 datafile 2023-12-02T01:33:28.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:28.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj85 datafile 2023-12-02T01:33:28.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:28.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj86 datafile 2023-12-02T01:33:28.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:28.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj87 datafile 2023-12-02T01:33:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:28.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj88 datafile 2023-12-02T01:33:28.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:28.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj89 datafile 2023-12-02T01:33:28.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:28.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj90 datafile 2023-12-02T01:33:28.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:28.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj91 datafile 2023-12-02T01:33:29.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:29.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj92 datafile 2023-12-02T01:33:29.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:29.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj93 datafile 2023-12-02T01:33:29.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:29.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj94 datafile 2023-12-02T01:33:29.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:29.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj95 datafile 2023-12-02T01:33:29.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:29.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj96 datafile 2023-12-02T01:33:29.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:29.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj97 datafile 2023-12-02T01:33:29.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:29.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj98 datafile 2023-12-02T01:33:30.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:30.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj99 datafile 2023-12-02T01:33:30.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:30.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj100 datafile 2023-12-02T01:33:30.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:30.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj101 datafile 2023-12-02T01:33:30.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:30.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj102 datafile 2023-12-02T01:33:30.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:30.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj103 datafile 2023-12-02T01:33:30.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:30.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj104 datafile 2023-12-02T01:33:30.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:30.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj105 datafile 2023-12-02T01:33:31.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:31.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj106 datafile 2023-12-02T01:33:31.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:31.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj107 datafile 2023-12-02T01:33:31.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:31.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj108 datafile 2023-12-02T01:33:31.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:31.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj109 datafile 2023-12-02T01:33:31.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:31.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj110 datafile 2023-12-02T01:33:31.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:31.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj111 datafile 2023-12-02T01:33:32.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:32.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj112 datafile 2023-12-02T01:33:32.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:32.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj113 datafile 2023-12-02T01:33:32.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:32.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj114 datafile 2023-12-02T01:33:32.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:32.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj115 datafile 2023-12-02T01:33:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj116 datafile 2023-12-02T01:33:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj117 datafile 2023-12-02T01:33:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj118 datafile 2023-12-02T01:33:33.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:33.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj119 datafile 2023-12-02T01:33:33.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:33.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj120 datafile 2023-12-02T01:33:33.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:33.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj121 datafile 2023-12-02T01:33:33.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj122 datafile 2023-12-02T01:33:33.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:33.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj123 datafile 2023-12-02T01:33:33.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:33.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj124 datafile 2023-12-02T01:33:33.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:33.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj125 datafile 2023-12-02T01:33:34.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:34.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj126 datafile 2023-12-02T01:33:34.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:34.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj127 datafile 2023-12-02T01:33:34.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:34.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj128 datafile 2023-12-02T01:33:34.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:34.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj129 datafile 2023-12-02T01:33:34.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:34.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj130 datafile 2023-12-02T01:33:34.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:34.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj131 datafile 2023-12-02T01:33:34.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:34.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj132 datafile 2023-12-02T01:33:35.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:35.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj133 datafile 2023-12-02T01:33:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj134 datafile 2023-12-02T01:33:35.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:35.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj135 datafile 2023-12-02T01:33:35.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:35.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj136 datafile 2023-12-02T01:33:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj137 datafile 2023-12-02T01:33:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:35.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj138 datafile 2023-12-02T01:33:35.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:35.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj139 datafile 2023-12-02T01:33:36.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:36.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj140 datafile 2023-12-02T01:33:36.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:36.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj141 datafile 2023-12-02T01:33:36.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:36.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj142 datafile 2023-12-02T01:33:36.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:36.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj143 datafile 2023-12-02T01:33:36.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:36.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj144 datafile 2023-12-02T01:33:36.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:36.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj145 datafile 2023-12-02T01:33:36.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:36.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj146 datafile 2023-12-02T01:33:37.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:37.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj147 datafile 2023-12-02T01:33:37.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:37.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj148 datafile 2023-12-02T01:33:37.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:37.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj149 datafile 2023-12-02T01:33:37.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:37.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj150 datafile 2023-12-02T01:33:38.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:38.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj151 datafile 2023-12-02T01:33:38.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:38.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj152 datafile 2023-12-02T01:33:38.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:38.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj153 datafile 2023-12-02T01:33:38.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:38.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj154 datafile 2023-12-02T01:33:38.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:38.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj155 datafile 2023-12-02T01:33:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj156 datafile 2023-12-02T01:33:38.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:38.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj157 datafile 2023-12-02T01:33:39.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:39.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj158 datafile 2023-12-02T01:33:39.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:39.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj159 datafile 2023-12-02T01:33:39.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:39.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj160 datafile 2023-12-02T01:33:39.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:39.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj161 datafile 2023-12-02T01:33:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:39.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj162 datafile 2023-12-02T01:33:39.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:39.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj163 datafile 2023-12-02T01:33:39.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:39.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj164 datafile 2023-12-02T01:33:40.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:40.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj165 datafile 2023-12-02T01:33:40.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:40.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj166 datafile 2023-12-02T01:33:40.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:40.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj167 datafile 2023-12-02T01:33:40.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:40.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj168 datafile 2023-12-02T01:33:40.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:40.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj169 datafile 2023-12-02T01:33:40.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:40.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj170 datafile 2023-12-02T01:33:40.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:40.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj171 datafile 2023-12-02T01:33:41.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:41.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj172 datafile 2023-12-02T01:33:41.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:41.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj173 datafile 2023-12-02T01:33:41.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:41.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj174 datafile 2023-12-02T01:33:41.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:41.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj175 datafile 2023-12-02T01:33:41.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:41.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj176 datafile 2023-12-02T01:33:41.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:41.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj177 datafile 2023-12-02T01:33:41.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:41.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj178 datafile 2023-12-02T01:33:42.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:42.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj179 datafile 2023-12-02T01:33:42.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:42.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj180 datafile 2023-12-02T01:33:42.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:42.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj181 datafile 2023-12-02T01:33:42.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:42.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj182 datafile 2023-12-02T01:33:42.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:42.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj183 datafile 2023-12-02T01:33:42.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:42.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj184 datafile 2023-12-02T01:33:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:42.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj185 datafile 2023-12-02T01:33:43.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:43.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj186 datafile 2023-12-02T01:33:43.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:43.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj187 datafile 2023-12-02T01:33:43.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:43.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj188 datafile 2023-12-02T01:33:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:43.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj189 datafile 2023-12-02T01:33:43.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:43.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj190 datafile 2023-12-02T01:33:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj191 datafile 2023-12-02T01:33:44.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:44.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj192 datafile 2023-12-02T01:33:44.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:44.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj193 datafile 2023-12-02T01:33:44.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:44.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj194 datafile 2023-12-02T01:33:44.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:44.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj195 datafile 2023-12-02T01:33:44.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:44.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj196 datafile 2023-12-02T01:33:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj197 datafile 2023-12-02T01:33:44.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:44.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj198 datafile 2023-12-02T01:33:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj199 datafile 2023-12-02T01:33:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj200 datafile 2023-12-02T01:33:45.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:45.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj201 datafile 2023-12-02T01:33:45.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:45.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj202 datafile 2023-12-02T01:33:45.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:45.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj203 datafile 2023-12-02T01:33:45.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:45.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj204 datafile 2023-12-02T01:33:45.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:45.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj205 datafile 2023-12-02T01:33:46.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj206 datafile 2023-12-02T01:33:46.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:46.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj207 datafile 2023-12-02T01:33:46.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:46.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj208 datafile 2023-12-02T01:33:46.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:46.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj209 datafile 2023-12-02T01:33:46.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:46.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj210 datafile 2023-12-02T01:33:46.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:46.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj211 datafile 2023-12-02T01:33:47.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:47.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj212 datafile 2023-12-02T01:33:47.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:47.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj213 datafile 2023-12-02T01:33:47.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:47.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj214 datafile 2023-12-02T01:33:47.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:47.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj215 datafile 2023-12-02T01:33:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj216 datafile 2023-12-02T01:33:47.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:47.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj217 datafile 2023-12-02T01:33:47.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:47.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj218 datafile 2023-12-02T01:33:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj219 datafile 2023-12-02T01:33:48.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:48.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj220 datafile 2023-12-02T01:33:48.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:48.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj221 datafile 2023-12-02T01:33:48.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:48.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj222 datafile 2023-12-02T01:33:48.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:48.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj223 datafile 2023-12-02T01:33:48.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:48.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj224 datafile 2023-12-02T01:33:48.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:48.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj225 datafile 2023-12-02T01:33:49.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:49.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj226 datafile 2023-12-02T01:33:49.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:49.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj227 datafile 2023-12-02T01:33:49.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:49.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj228 datafile 2023-12-02T01:33:49.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:49.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj229 datafile 2023-12-02T01:33:49.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:49.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj230 datafile 2023-12-02T01:33:49.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:49.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj231 datafile 2023-12-02T01:33:49.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:49.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj232 datafile 2023-12-02T01:33:50.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:50.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj233 datafile 2023-12-02T01:33:50.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:50.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj234 datafile 2023-12-02T01:33:50.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:50.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj235 datafile 2023-12-02T01:33:50.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:50.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj236 datafile 2023-12-02T01:33:50.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:50.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj237 datafile 2023-12-02T01:33:50.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:50.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj238 datafile 2023-12-02T01:33:50.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:50.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj239 datafile 2023-12-02T01:33:51.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:51.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj240 datafile 2023-12-02T01:33:51.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:51.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj241 datafile 2023-12-02T01:33:51.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:51.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj242 datafile 2023-12-02T01:33:51.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:51.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj243 datafile 2023-12-02T01:33:51.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:51.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj244 datafile 2023-12-02T01:33:51.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:51.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj245 datafile 2023-12-02T01:33:51.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:51.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj246 datafile 2023-12-02T01:33:52.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:52.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj247 datafile 2023-12-02T01:33:52.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:52.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj248 datafile 2023-12-02T01:33:52.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:52.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj249 datafile 2023-12-02T01:33:52.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:52.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj250 datafile 2023-12-02T01:33:52.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:52.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj251 datafile 2023-12-02T01:33:52.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:52.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj252 datafile 2023-12-02T01:33:53.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:53.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj253 datafile 2023-12-02T01:33:53.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:53.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj254 datafile 2023-12-02T01:33:53.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:53.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj255 datafile 2023-12-02T01:33:53.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:53.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj256 datafile 2023-12-02T01:33:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj257 datafile 2023-12-02T01:33:53.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:53.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj258 datafile 2023-12-02T01:33:53.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:53.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj259 datafile 2023-12-02T01:33:54.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:54.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj260 datafile 2023-12-02T01:33:54.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:54.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj261 datafile 2023-12-02T01:33:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj262 datafile 2023-12-02T01:33:54.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:54.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj263 datafile 2023-12-02T01:33:54.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:54.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj264 datafile 2023-12-02T01:33:54.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:54.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj265 datafile 2023-12-02T01:33:54.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:54.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj266 datafile 2023-12-02T01:33:55.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:55.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj267 datafile 2023-12-02T01:33:55.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:55.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj268 datafile 2023-12-02T01:33:55.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:55.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj269 datafile 2023-12-02T01:33:55.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:55.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj270 datafile 2023-12-02T01:33:55.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:55.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj271 datafile 2023-12-02T01:33:55.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:55.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj272 datafile 2023-12-02T01:33:56.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:56.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj273 datafile 2023-12-02T01:33:56.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:56.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj274 datafile 2023-12-02T01:33:56.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:56.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj275 datafile 2023-12-02T01:33:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj276 datafile 2023-12-02T01:33:56.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:56.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj277 datafile 2023-12-02T01:33:56.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:56.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj278 datafile 2023-12-02T01:33:56.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:56.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj279 datafile 2023-12-02T01:33:57.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:57.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj280 datafile 2023-12-02T01:33:57.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:57.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj281 datafile 2023-12-02T01:33:57.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:57.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj282 datafile 2023-12-02T01:33:57.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:57.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj283 datafile 2023-12-02T01:33:57.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:57.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj284 datafile 2023-12-02T01:33:57.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:57.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj285 datafile 2023-12-02T01:33:57.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:57.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj286 datafile 2023-12-02T01:33:58.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:58.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj287 datafile 2023-12-02T01:33:58.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:58.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj288 datafile 2023-12-02T01:33:58.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:58.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj289 datafile 2023-12-02T01:33:58.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:58.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj290 datafile 2023-12-02T01:33:58.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:58.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj291 datafile 2023-12-02T01:33:58.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:58.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj292 datafile 2023-12-02T01:33:58.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:58.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj293 datafile 2023-12-02T01:33:59.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:59.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj294 datafile 2023-12-02T01:33:59.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:59.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj295 datafile 2023-12-02T01:33:59.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:59.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj296 datafile 2023-12-02T01:33:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj297 datafile 2023-12-02T01:33:59.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:59.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj298 datafile 2023-12-02T01:33:59.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:33:59.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj299 datafile 2023-12-02T01:34:00.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:00.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj300 datafile 2023-12-02T01:34:00.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:00.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj301 datafile 2023-12-02T01:34:00.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:00.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj302 datafile 2023-12-02T01:34:00.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:00.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj303 datafile 2023-12-02T01:34:00.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:00.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj304 datafile 2023-12-02T01:34:00.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:00.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj305 datafile 2023-12-02T01:34:00.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:00.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj306 datafile 2023-12-02T01:34:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:01.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj307 datafile 2023-12-02T01:34:01.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:01.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj308 datafile 2023-12-02T01:34:01.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:01.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj309 datafile 2023-12-02T01:34:01.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:01.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj310 datafile 2023-12-02T01:34:01.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:01.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj311 datafile 2023-12-02T01:34:01.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:01.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj312 datafile 2023-12-02T01:34:01.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:01.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj313 datafile 2023-12-02T01:34:02.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:02.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj314 datafile 2023-12-02T01:34:02.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:02.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj315 datafile 2023-12-02T01:34:02.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:02.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj316 datafile 2023-12-02T01:34:02.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:02.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj317 datafile 2023-12-02T01:34:02.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:02.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj318 datafile 2023-12-02T01:34:02.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:02.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj319 datafile 2023-12-02T01:34:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj320 datafile 2023-12-02T01:34:03.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:03.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj321 datafile 2023-12-02T01:34:03.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:03.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj322 datafile 2023-12-02T01:34:03.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:03.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj323 datafile 2023-12-02T01:34:03.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:03.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj324 datafile 2023-12-02T01:34:03.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:03.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj325 datafile 2023-12-02T01:34:03.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:03.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj326 datafile 2023-12-02T01:34:03.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:03.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj327 datafile 2023-12-02T01:34:04.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:04.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj328 datafile 2023-12-02T01:34:04.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:04.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj329 datafile 2023-12-02T01:34:04.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:04.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj330 datafile 2023-12-02T01:34:04.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:04.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj331 datafile 2023-12-02T01:34:04.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:04.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj332 datafile 2023-12-02T01:34:04.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:04.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj333 datafile 2023-12-02T01:34:05.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:05.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj334 datafile 2023-12-02T01:34:05.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:05.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj335 datafile 2023-12-02T01:34:05.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:05.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj336 datafile 2023-12-02T01:34:05.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:05.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj337 datafile 2023-12-02T01:34:05.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:05.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj338 datafile 2023-12-02T01:34:05.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:05.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj339 datafile 2023-12-02T01:34:05.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:05.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj340 datafile 2023-12-02T01:34:06.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:06.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj341 datafile 2023-12-02T01:34:06.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:06.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj342 datafile 2023-12-02T01:34:06.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:06.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj343 datafile 2023-12-02T01:34:06.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:06.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj344 datafile 2023-12-02T01:34:06.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:06.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj345 datafile 2023-12-02T01:34:06.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:06.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj346 datafile 2023-12-02T01:34:07.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:07.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj347 datafile 2023-12-02T01:34:07.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:07.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj348 datafile 2023-12-02T01:34:07.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:07.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj349 datafile 2023-12-02T01:34:07.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:07.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj350 datafile 2023-12-02T01:34:07.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:07.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj351 datafile 2023-12-02T01:34:07.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:07.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj352 datafile 2023-12-02T01:34:07.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:07.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj353 datafile 2023-12-02T01:34:08.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:08.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj354 datafile 2023-12-02T01:34:08.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:08.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj355 datafile 2023-12-02T01:34:08.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:08.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj356 datafile 2023-12-02T01:34:08.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:08.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj357 datafile 2023-12-02T01:34:08.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:08.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj358 datafile 2023-12-02T01:34:09.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:09.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj359 datafile 2023-12-02T01:34:09.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:09.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj360 datafile 2023-12-02T01:34:09.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:09.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj361 datafile 2023-12-02T01:34:09.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:09.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj362 datafile 2023-12-02T01:34:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj363 datafile 2023-12-02T01:34:09.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:09.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj364 datafile 2023-12-02T01:34:09.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:09.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj365 datafile 2023-12-02T01:34:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj366 datafile 2023-12-02T01:34:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj367 datafile 2023-12-02T01:34:10.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:10.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj368 datafile 2023-12-02T01:34:10.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:10.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj369 datafile 2023-12-02T01:34:10.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:10.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj370 datafile 2023-12-02T01:34:10.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:10.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj371 datafile 2023-12-02T01:34:10.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:10.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj372 datafile 2023-12-02T01:34:11.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:11.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj373 datafile 2023-12-02T01:34:11.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:11.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj374 datafile 2023-12-02T01:34:11.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:11.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj375 datafile 2023-12-02T01:34:11.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:11.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj376 datafile 2023-12-02T01:34:11.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:11.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj377 datafile 2023-12-02T01:34:11.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:11.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj378 datafile 2023-12-02T01:34:11.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:11.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj379 datafile 2023-12-02T01:34:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:12.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj380 datafile 2023-12-02T01:34:12.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:12.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj381 datafile 2023-12-02T01:34:12.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:12.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj382 datafile 2023-12-02T01:34:12.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:12.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj383 datafile 2023-12-02T01:34:12.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:12.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj384 datafile 2023-12-02T01:34:12.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:12.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj385 datafile 2023-12-02T01:34:12.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:12.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj386 datafile 2023-12-02T01:34:13.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:13.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj387 datafile 2023-12-02T01:34:13.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:13.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj388 datafile 2023-12-02T01:34:13.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:13.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj389 datafile 2023-12-02T01:34:13.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:13.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj390 datafile 2023-12-02T01:34:13.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:13.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj391 datafile 2023-12-02T01:34:13.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:13.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj392 datafile 2023-12-02T01:34:14.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:14.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj393 datafile 2023-12-02T01:34:14.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:14.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj394 datafile 2023-12-02T01:34:14.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:14.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj395 datafile 2023-12-02T01:34:14.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:14.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj396 datafile 2023-12-02T01:34:14.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:14.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj397 datafile 2023-12-02T01:34:14.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:14.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj398 datafile 2023-12-02T01:34:14.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:14.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj399 datafile 2023-12-02T01:34:15.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:15.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj400 datafile 2023-12-02T01:34:15.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:15.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj401 datafile 2023-12-02T01:34:15.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:15.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj402 datafile 2023-12-02T01:34:15.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:15.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj403 datafile 2023-12-02T01:34:15.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:15.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj404 datafile 2023-12-02T01:34:15.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:15.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj405 datafile 2023-12-02T01:34:15.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:15.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj406 datafile 2023-12-02T01:34:16.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:16.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj407 datafile 2023-12-02T01:34:16.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:16.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj408 datafile 2023-12-02T01:34:16.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:16.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj409 datafile 2023-12-02T01:34:16.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:16.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj410 datafile 2023-12-02T01:34:16.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:16.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj411 datafile 2023-12-02T01:34:16.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:16.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj412 datafile 2023-12-02T01:34:17.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:17.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj413 datafile 2023-12-02T01:34:17.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:17.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj414 datafile 2023-12-02T01:34:17.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:17.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj415 datafile 2023-12-02T01:34:17.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:17.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj416 datafile 2023-12-02T01:34:17.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:17.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj417 datafile 2023-12-02T01:34:17.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:17.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj418 datafile 2023-12-02T01:34:17.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj419 datafile 2023-12-02T01:34:18.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj420 datafile 2023-12-02T01:34:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj421 datafile 2023-12-02T01:34:18.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj422 datafile 2023-12-02T01:34:18.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj423 datafile 2023-12-02T01:34:18.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj424 datafile 2023-12-02T01:34:18.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj425 datafile 2023-12-02T01:34:18.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj426 datafile 2023-12-02T01:34:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj427 datafile 2023-12-02T01:34:19.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:19.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj428 datafile 2023-12-02T01:34:19.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:19.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj429 datafile 2023-12-02T01:34:19.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:19.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj430 datafile 2023-12-02T01:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj431 datafile 2023-12-02T01:34:19.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:19.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj432 datafile 2023-12-02T01:34:19.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:19.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj433 datafile 2023-12-02T01:34:20.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:20.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj434 datafile 2023-12-02T01:34:20.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:20.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj435 datafile 2023-12-02T01:34:20.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:20.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj436 datafile 2023-12-02T01:34:20.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:20.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj437 datafile 2023-12-02T01:34:20.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:20.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj438 datafile 2023-12-02T01:34:20.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:20.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj439 datafile 2023-12-02T01:34:20.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:20.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj440 datafile 2023-12-02T01:34:21.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:21.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj441 datafile 2023-12-02T01:34:21.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:21.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj442 datafile 2023-12-02T01:34:21.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:21.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj443 datafile 2023-12-02T01:34:21.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:21.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj444 datafile 2023-12-02T01:34:21.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:21.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj445 datafile 2023-12-02T01:34:21.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:21.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj446 datafile 2023-12-02T01:34:21.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:21.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj447 datafile 2023-12-02T01:34:22.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:22.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj448 datafile 2023-12-02T01:34:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj449 datafile 2023-12-02T01:34:22.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:22.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj450 datafile 2023-12-02T01:34:22.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:22.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj451 datafile 2023-12-02T01:34:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj452 datafile 2023-12-02T01:34:22.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:22.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj453 datafile 2023-12-02T01:34:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj454 datafile 2023-12-02T01:34:23.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:23.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj455 datafile 2023-12-02T01:34:23.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:23.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj456 datafile 2023-12-02T01:34:23.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:23.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj457 datafile 2023-12-02T01:34:23.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:23.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj458 datafile 2023-12-02T01:34:23.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:23.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj459 datafile 2023-12-02T01:34:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj460 datafile 2023-12-02T01:34:24.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:24.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj461 datafile 2023-12-02T01:34:24.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:24.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj462 datafile 2023-12-02T01:34:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj463 datafile 2023-12-02T01:34:24.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:24.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj464 datafile 2023-12-02T01:34:24.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:24.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj465 datafile 2023-12-02T01:34:24.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:24.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj466 datafile 2023-12-02T01:34:25.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:25.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj467 datafile 2023-12-02T01:34:25.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:25.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj468 datafile 2023-12-02T01:34:25.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:25.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj469 datafile 2023-12-02T01:34:25.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:25.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj470 datafile 2023-12-02T01:34:25.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:25.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj471 datafile 2023-12-02T01:34:25.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:25.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj472 datafile 2023-12-02T01:34:25.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:25.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj473 datafile 2023-12-02T01:34:26.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:26.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj474 datafile 2023-12-02T01:34:26.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:26.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj475 datafile 2023-12-02T01:34:26.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:26.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj476 datafile 2023-12-02T01:34:26.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:26.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj477 datafile 2023-12-02T01:34:26.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:26.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj478 datafile 2023-12-02T01:34:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:26.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj479 datafile 2023-12-02T01:34:27.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:27.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj480 datafile 2023-12-02T01:34:27.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:27.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj481 datafile 2023-12-02T01:34:27.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:27.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj482 datafile 2023-12-02T01:34:27.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:27.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj483 datafile 2023-12-02T01:34:27.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:27.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj484 datafile 2023-12-02T01:34:27.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:27.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj485 datafile 2023-12-02T01:34:27.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:27.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj486 datafile 2023-12-02T01:34:28.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:28.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj487 datafile 2023-12-02T01:34:28.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:28.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj488 datafile 2023-12-02T01:34:28.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:28.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj489 datafile 2023-12-02T01:34:28.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:28.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj490 datafile 2023-12-02T01:34:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj491 datafile 2023-12-02T01:34:28.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:28.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj492 datafile 2023-12-02T01:34:28.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:28.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj493 datafile 2023-12-02T01:34:29.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:29.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj494 datafile 2023-12-02T01:34:29.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:29.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj495 datafile 2023-12-02T01:34:29.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:29.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj496 datafile 2023-12-02T01:34:29.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:29.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj497 datafile 2023-12-02T01:34:29.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:29.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj498 datafile 2023-12-02T01:34:29.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:29.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj499 datafile 2023-12-02T01:34:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj500 datafile 2023-12-02T01:34:30.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:30.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj501 datafile 2023-12-02T01:34:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj502 datafile 2023-12-02T01:34:30.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:30.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj503 datafile 2023-12-02T01:34:30.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:30.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj504 datafile 2023-12-02T01:34:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:30.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj505 datafile 2023-12-02T01:34:30.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:30.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj506 datafile 2023-12-02T01:34:30.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:30.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj507 datafile 2023-12-02T01:34:31.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:31.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj508 datafile 2023-12-02T01:34:31.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:31.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj509 datafile 2023-12-02T01:34:31.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:31.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj510 datafile 2023-12-02T01:34:31.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:31.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj511 datafile 2023-12-02T01:34:31.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:31.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj512 datafile 2023-12-02T01:34:31.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:31.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj513 datafile 2023-12-02T01:34:31.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:31.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj514 datafile 2023-12-02T01:34:32.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:32.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj515 datafile 2023-12-02T01:34:32.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:32.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj516 datafile 2023-12-02T01:34:32.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:32.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj517 datafile 2023-12-02T01:34:32.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:32.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj518 datafile 2023-12-02T01:34:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj519 datafile 2023-12-02T01:34:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj520 datafile 2023-12-02T01:34:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj521 datafile 2023-12-02T01:34:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj522 datafile 2023-12-02T01:34:33.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:33.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj523 datafile 2023-12-02T01:34:33.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj524 datafile 2023-12-02T01:34:33.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:33.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj525 datafile 2023-12-02T01:34:33.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:33.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj526 datafile 2023-12-02T01:34:33.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:33.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj527 datafile 2023-12-02T01:34:33.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:33.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj528 datafile 2023-12-02T01:34:34.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:34.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj529 datafile 2023-12-02T01:34:34.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:34.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj530 datafile 2023-12-02T01:34:34.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:34.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj531 datafile 2023-12-02T01:34:34.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:34.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj532 datafile 2023-12-02T01:34:34.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:34.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj533 datafile 2023-12-02T01:34:34.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:34.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj534 datafile 2023-12-02T01:34:35.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:35.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj535 datafile 2023-12-02T01:34:35.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:35.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj536 datafile 2023-12-02T01:34:35.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:35.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj537 datafile 2023-12-02T01:34:35.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:35.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj538 datafile 2023-12-02T01:34:35.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:35.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj539 datafile 2023-12-02T01:34:35.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:35.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj540 datafile 2023-12-02T01:34:35.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:35.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj541 datafile 2023-12-02T01:34:36.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:36.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj542 datafile 2023-12-02T01:34:36.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:36.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj543 datafile 2023-12-02T01:34:36.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:36.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj544 datafile 2023-12-02T01:34:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj545 datafile 2023-12-02T01:34:36.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:36.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj546 datafile 2023-12-02T01:34:36.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:36.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj547 datafile 2023-12-02T01:34:36.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:36.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj548 datafile 2023-12-02T01:34:37.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:37.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj549 datafile 2023-12-02T01:34:37.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:37.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj550 datafile 2023-12-02T01:34:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:37.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj551 datafile 2023-12-02T01:34:37.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:37.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj552 datafile 2023-12-02T01:34:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj553 datafile 2023-12-02T01:34:37.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:37.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj554 datafile 2023-12-02T01:34:38.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:38.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj555 datafile 2023-12-02T01:34:38.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:38.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj556 datafile 2023-12-02T01:34:38.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:38.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj557 datafile 2023-12-02T01:34:38.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:38.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj558 datafile 2023-12-02T01:34:38.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:38.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj559 datafile 2023-12-02T01:34:38.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:38.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj560 datafile 2023-12-02T01:34:38.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:38.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj561 datafile 2023-12-02T01:34:39.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:39.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj562 datafile 2023-12-02T01:34:39.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:39.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj563 datafile 2023-12-02T01:34:39.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:39.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj564 datafile 2023-12-02T01:34:39.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:39.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj565 datafile 2023-12-02T01:34:39.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:39.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj566 datafile 2023-12-02T01:34:40.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:40.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj567 datafile 2023-12-02T01:34:40.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:40.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj568 datafile 2023-12-02T01:34:40.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:40.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj569 datafile 2023-12-02T01:34:40.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:40.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj570 datafile 2023-12-02T01:34:40.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:40.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj571 datafile 2023-12-02T01:34:40.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:40.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj572 datafile 2023-12-02T01:34:41.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:41.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj573 datafile 2023-12-02T01:34:41.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:41.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj574 datafile 2023-12-02T01:34:41.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:41.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj575 datafile 2023-12-02T01:34:41.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:41.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj576 datafile 2023-12-02T01:34:41.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:41.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj577 datafile 2023-12-02T01:34:41.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:41.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj578 datafile 2023-12-02T01:34:41.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:41.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj579 datafile 2023-12-02T01:34:42.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:42.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj580 datafile 2023-12-02T01:34:42.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:42.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj581 datafile 2023-12-02T01:34:42.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:42.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj582 datafile 2023-12-02T01:34:42.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:42.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj583 datafile 2023-12-02T01:34:42.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:42.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj584 datafile 2023-12-02T01:34:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj585 datafile 2023-12-02T01:34:43.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:43.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj586 datafile 2023-12-02T01:34:43.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:43.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj587 datafile 2023-12-02T01:34:43.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:43.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj588 datafile 2023-12-02T01:34:43.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:43.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj589 datafile 2023-12-02T01:34:43.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:43.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj590 datafile 2023-12-02T01:34:43.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:43.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj591 datafile 2023-12-02T01:34:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj592 datafile 2023-12-02T01:34:44.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:44.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj593 datafile 2023-12-02T01:34:44.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:44.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj594 datafile 2023-12-02T01:34:44.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:44.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj595 datafile 2023-12-02T01:34:44.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:44.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj596 datafile 2023-12-02T01:34:44.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:44.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj597 datafile 2023-12-02T01:34:44.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:44.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj598 datafile 2023-12-02T01:34:45.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:45.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj599 datafile 2023-12-02T01:34:45.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:45.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj600 datafile 2023-12-02T01:34:45.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:45.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj601 datafile 2023-12-02T01:34:45.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:45.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj602 datafile 2023-12-02T01:34:45.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:45.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj603 datafile 2023-12-02T01:34:45.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:45.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj604 datafile 2023-12-02T01:34:45.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:45.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj605 datafile 2023-12-02T01:34:46.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:46.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj606 datafile 2023-12-02T01:34:46.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:46.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj607 datafile 2023-12-02T01:34:46.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj608 datafile 2023-12-02T01:34:46.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:46.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj609 datafile 2023-12-02T01:34:46.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:46.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj610 datafile 2023-12-02T01:34:46.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:46.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj611 datafile 2023-12-02T01:34:46.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:46.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj612 datafile 2023-12-02T01:34:47.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:47.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj613 datafile 2023-12-02T01:34:47.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:47.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj614 datafile 2023-12-02T01:34:47.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:47.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj615 datafile 2023-12-02T01:34:47.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:47.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj616 datafile 2023-12-02T01:34:47.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:47.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj617 datafile 2023-12-02T01:34:47.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:47.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj618 datafile 2023-12-02T01:34:47.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:47.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj619 datafile 2023-12-02T01:34:48.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:48.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj620 datafile 2023-12-02T01:34:48.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:48.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj621 datafile 2023-12-02T01:34:48.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:48.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj622 datafile 2023-12-02T01:34:48.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:48.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj623 datafile 2023-12-02T01:34:48.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:48.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj624 datafile 2023-12-02T01:34:48.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:48.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj625 datafile 2023-12-02T01:34:49.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:49.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj626 datafile 2023-12-02T01:34:49.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:49.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj627 datafile 2023-12-02T01:34:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj628 datafile 2023-12-02T01:34:49.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:49.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj629 datafile 2023-12-02T01:34:49.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:49.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj630 datafile 2023-12-02T01:34:49.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:49.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj631 datafile 2023-12-02T01:34:49.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:49.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj632 datafile 2023-12-02T01:34:50.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:50.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj633 datafile 2023-12-02T01:34:50.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:50.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj634 datafile 2023-12-02T01:34:50.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:50.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj635 datafile 2023-12-02T01:34:50.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:50.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj636 datafile 2023-12-02T01:34:50.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:50.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj637 datafile 2023-12-02T01:34:50.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:50.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj638 datafile 2023-12-02T01:34:50.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:50.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj639 datafile 2023-12-02T01:34:51.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:51.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj640 datafile 2023-12-02T01:34:51.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:51.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj641 datafile 2023-12-02T01:34:51.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:51.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj642 datafile 2023-12-02T01:34:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj643 datafile 2023-12-02T01:34:51.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:51.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj644 datafile 2023-12-02T01:34:51.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:51.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj645 datafile 2023-12-02T01:34:52.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:52.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj646 datafile 2023-12-02T01:34:52.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:52.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj647 datafile 2023-12-02T01:34:52.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:52.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj648 datafile 2023-12-02T01:34:52.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:52.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj649 datafile 2023-12-02T01:34:52.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:52.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj650 datafile 2023-12-02T01:34:52.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:52.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj651 datafile 2023-12-02T01:34:52.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:52.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj652 datafile 2023-12-02T01:34:53.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:53.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj653 datafile 2023-12-02T01:34:53.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:53.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj654 datafile 2023-12-02T01:34:53.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:53.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj655 datafile 2023-12-02T01:34:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:53.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj656 datafile 2023-12-02T01:34:53.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:53.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj657 datafile 2023-12-02T01:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj658 datafile 2023-12-02T01:34:53.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:53.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj659 datafile 2023-12-02T01:34:54.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:54.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj660 datafile 2023-12-02T01:34:54.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:54.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj661 datafile 2023-12-02T01:34:54.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:54.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj662 datafile 2023-12-02T01:34:54.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:54.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj663 datafile 2023-12-02T01:34:54.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:54.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj664 datafile 2023-12-02T01:34:54.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:54.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj665 datafile 2023-12-02T01:34:55.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:55.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj666 datafile 2023-12-02T01:34:55.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:55.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj667 datafile 2023-12-02T01:34:55.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:55.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj668 datafile 2023-12-02T01:34:55.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:55.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj669 datafile 2023-12-02T01:34:55.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:55.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj670 datafile 2023-12-02T01:34:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj671 datafile 2023-12-02T01:34:55.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:55.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj672 datafile 2023-12-02T01:34:56.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:56.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj673 datafile 2023-12-02T01:34:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:56.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj674 datafile 2023-12-02T01:34:56.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:56.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj675 datafile 2023-12-02T01:34:56.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:56.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj676 datafile 2023-12-02T01:34:56.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:56.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj677 datafile 2023-12-02T01:34:56.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:56.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj678 datafile 2023-12-02T01:34:57.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:57.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj679 datafile 2023-12-02T01:34:57.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:57.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj680 datafile 2023-12-02T01:34:57.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:57.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj681 datafile 2023-12-02T01:34:57.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:57.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj682 datafile 2023-12-02T01:34:57.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:57.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj683 datafile 2023-12-02T01:34:57.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:57.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj684 datafile 2023-12-02T01:34:57.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:57.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj685 datafile 2023-12-02T01:34:58.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:58.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj686 datafile 2023-12-02T01:34:58.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:58.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj687 datafile 2023-12-02T01:34:58.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:58.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj688 datafile 2023-12-02T01:34:58.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:58.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj689 datafile 2023-12-02T01:34:58.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:58.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj690 datafile 2023-12-02T01:34:58.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:58.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj691 datafile 2023-12-02T01:34:58.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:58.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj692 datafile 2023-12-02T01:34:59.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:59.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj693 datafile 2023-12-02T01:34:59.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:59.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj694 datafile 2023-12-02T01:34:59.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:59.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj695 datafile 2023-12-02T01:34:59.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:59.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj696 datafile 2023-12-02T01:34:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj697 datafile 2023-12-02T01:34:59.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:34:59.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj698 datafile 2023-12-02T01:35:00.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:00.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj699 datafile 2023-12-02T01:35:00.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:00.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj700 datafile 2023-12-02T01:35:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj701 datafile 2023-12-02T01:35:00.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:00.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj702 datafile 2023-12-02T01:35:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:00.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj703 datafile 2023-12-02T01:35:00.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:00.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj704 datafile 2023-12-02T01:35:00.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:00.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj705 datafile 2023-12-02T01:35:01.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:01.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj706 datafile 2023-12-02T01:35:01.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:01.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj707 datafile 2023-12-02T01:35:01.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:01.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj708 datafile 2023-12-02T01:35:01.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:01.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj709 datafile 2023-12-02T01:35:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj710 datafile 2023-12-02T01:35:01.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:01.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj711 datafile 2023-12-02T01:35:01.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:01.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj712 datafile 2023-12-02T01:35:02.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:02.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj713 datafile 2023-12-02T01:35:02.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:02.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj714 datafile 2023-12-02T01:35:02.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:02.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj715 datafile 2023-12-02T01:35:02.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:02.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj716 datafile 2023-12-02T01:35:02.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:02.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj717 datafile 2023-12-02T01:35:02.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:02.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj718 datafile 2023-12-02T01:35:02.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:02.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj719 datafile 2023-12-02T01:35:03.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:03.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj720 datafile 2023-12-02T01:35:03.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:03.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj721 datafile 2023-12-02T01:35:03.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:03.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj722 datafile 2023-12-02T01:35:03.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:03.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj723 datafile 2023-12-02T01:35:03.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:03.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj724 datafile 2023-12-02T01:35:03.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:03.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj725 datafile 2023-12-02T01:35:03.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:03.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj726 datafile 2023-12-02T01:35:04.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:04.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj727 datafile 2023-12-02T01:35:04.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:04.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj728 datafile 2023-12-02T01:35:04.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:04.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj729 datafile 2023-12-02T01:35:04.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:04.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj730 datafile 2023-12-02T01:35:04.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:04.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj731 datafile 2023-12-02T01:35:04.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:04.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj732 datafile 2023-12-02T01:35:04.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:04.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj733 datafile 2023-12-02T01:35:05.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:05.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj734 datafile 2023-12-02T01:35:05.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:05.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj735 datafile 2023-12-02T01:35:05.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:05.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj736 datafile 2023-12-02T01:35:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj737 datafile 2023-12-02T01:35:05.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:05.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj738 datafile 2023-12-02T01:35:05.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:05.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj739 datafile 2023-12-02T01:35:06.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj740 datafile 2023-12-02T01:35:06.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj741 datafile 2023-12-02T01:35:06.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj742 datafile 2023-12-02T01:35:06.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj743 datafile 2023-12-02T01:35:06.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj744 datafile 2023-12-02T01:35:06.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj745 datafile 2023-12-02T01:35:06.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj746 datafile 2023-12-02T01:35:06.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:06.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj747 datafile 2023-12-02T01:35:07.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:07.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj748 datafile 2023-12-02T01:35:07.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:07.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj749 datafile 2023-12-02T01:35:07.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:07.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj750 datafile 2023-12-02T01:35:07.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:07.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj751 datafile 2023-12-02T01:35:07.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:07.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj752 datafile 2023-12-02T01:35:07.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:07.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj753 datafile 2023-12-02T01:35:08.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:08.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj754 datafile 2023-12-02T01:35:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj755 datafile 2023-12-02T01:35:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj756 datafile 2023-12-02T01:35:08.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:08.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj757 datafile 2023-12-02T01:35:08.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:08.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj758 datafile 2023-12-02T01:35:08.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:08.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj759 datafile 2023-12-02T01:35:08.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:08.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj760 datafile 2023-12-02T01:35:09.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:09.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj761 datafile 2023-12-02T01:35:09.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:09.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj762 datafile 2023-12-02T01:35:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:09.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj763 datafile 2023-12-02T01:35:09.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:09.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj764 datafile 2023-12-02T01:35:09.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:09.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj765 datafile 2023-12-02T01:35:09.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:09.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj766 datafile 2023-12-02T01:35:09.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:09.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj767 datafile 2023-12-02T01:35:10.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:10.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj768 datafile 2023-12-02T01:35:10.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:10.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj769 datafile 2023-12-02T01:35:10.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:10.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj770 datafile 2023-12-02T01:35:10.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:10.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj771 datafile 2023-12-02T01:35:10.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:10.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj772 datafile 2023-12-02T01:35:11.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:11.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj773 datafile 2023-12-02T01:35:11.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:11.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj774 datafile 2023-12-02T01:35:11.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:11.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj775 datafile 2023-12-02T01:35:11.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:11.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj776 datafile 2023-12-02T01:35:11.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:11.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj777 datafile 2023-12-02T01:35:11.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:11.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj778 datafile 2023-12-02T01:35:12.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:12.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj779 datafile 2023-12-02T01:35:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj780 datafile 2023-12-02T01:35:12.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:12.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj781 datafile 2023-12-02T01:35:12.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:12.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj782 datafile 2023-12-02T01:35:12.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:12.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj783 datafile 2023-12-02T01:35:12.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:12.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj784 datafile 2023-12-02T01:35:12.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:12.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj785 datafile 2023-12-02T01:35:13.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj786 datafile 2023-12-02T01:35:13.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:13.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj787 datafile 2023-12-02T01:35:13.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:13.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj788 datafile 2023-12-02T01:35:13.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:13.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj789 datafile 2023-12-02T01:35:13.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:13.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj790 datafile 2023-12-02T01:35:13.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:13.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj791 datafile 2023-12-02T01:35:13.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:13.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj792 datafile 2023-12-02T01:35:14.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:14.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj793 datafile 2023-12-02T01:35:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:14.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj794 datafile 2023-12-02T01:35:14.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:14.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj795 datafile 2023-12-02T01:35:14.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:14.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj796 datafile 2023-12-02T01:35:14.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:14.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj797 datafile 2023-12-02T01:35:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj798 datafile 2023-12-02T01:35:14.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:14.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj799 datafile 2023-12-02T01:35:15.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:15.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj800 datafile 2023-12-02T01:35:15.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:15.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj801 datafile 2023-12-02T01:35:15.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:15.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj802 datafile 2023-12-02T01:35:15.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:15.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj803 datafile 2023-12-02T01:35:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj804 datafile 2023-12-02T01:35:15.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:15.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj805 datafile 2023-12-02T01:35:15.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:15.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj806 datafile 2023-12-02T01:35:16.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:16.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj807 datafile 2023-12-02T01:35:16.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:16.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj808 datafile 2023-12-02T01:35:16.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:16.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj809 datafile 2023-12-02T01:35:16.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:16.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj810 datafile 2023-12-02T01:35:16.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:16.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj811 datafile 2023-12-02T01:35:16.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:16.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj812 datafile 2023-12-02T01:35:17.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:17.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj813 datafile 2023-12-02T01:35:17.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:17.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj814 datafile 2023-12-02T01:35:17.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:17.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj815 datafile 2023-12-02T01:35:17.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:17.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj816 datafile 2023-12-02T01:35:17.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:17.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj817 datafile 2023-12-02T01:35:17.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:17.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj818 datafile 2023-12-02T01:35:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj819 datafile 2023-12-02T01:35:18.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:18.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj820 datafile 2023-12-02T01:35:18.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:18.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj821 datafile 2023-12-02T01:35:18.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:18.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj822 datafile 2023-12-02T01:35:18.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:18.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj823 datafile 2023-12-02T01:35:18.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:18.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj824 datafile 2023-12-02T01:35:18.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:18.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj825 datafile 2023-12-02T01:35:18.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:18.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj826 datafile 2023-12-02T01:35:19.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:19.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj827 datafile 2023-12-02T01:35:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj828 datafile 2023-12-02T01:35:19.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:19.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj829 datafile 2023-12-02T01:35:19.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:19.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj830 datafile 2023-12-02T01:35:19.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:19.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj831 datafile 2023-12-02T01:35:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:19.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj832 datafile 2023-12-02T01:35:20.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:20.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj833 datafile 2023-12-02T01:35:20.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:20.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj834 datafile 2023-12-02T01:35:20.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:20.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj835 datafile 2023-12-02T01:35:20.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:20.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj836 datafile 2023-12-02T01:35:20.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:20.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj837 datafile 2023-12-02T01:35:20.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj838 datafile 2023-12-02T01:35:20.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:20.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj839 datafile 2023-12-02T01:35:21.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:21.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj840 datafile 2023-12-02T01:35:21.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:21.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj841 datafile 2023-12-02T01:35:21.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:21.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj842 datafile 2023-12-02T01:35:21.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:21.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj843 datafile 2023-12-02T01:35:21.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:21.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj844 datafile 2023-12-02T01:35:21.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:21.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj845 datafile 2023-12-02T01:35:21.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:22.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj846 datafile 2023-12-02T01:35:22.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:22.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj847 datafile 2023-12-02T01:35:22.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:22.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj848 datafile 2023-12-02T01:35:22.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:22.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj849 datafile 2023-12-02T01:35:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj850 datafile 2023-12-02T01:35:22.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:22.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj851 datafile 2023-12-02T01:35:22.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:22.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj852 datafile 2023-12-02T01:35:23.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:23.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj853 datafile 2023-12-02T01:35:23.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:23.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj854 datafile 2023-12-02T01:35:23.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:23.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj855 datafile 2023-12-02T01:35:23.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:23.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj856 datafile 2023-12-02T01:35:23.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:23.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj857 datafile 2023-12-02T01:35:23.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:23.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj858 datafile 2023-12-02T01:35:23.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:23.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj859 datafile 2023-12-02T01:35:24.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:24.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj860 datafile 2023-12-02T01:35:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj861 datafile 2023-12-02T01:35:24.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:24.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj862 datafile 2023-12-02T01:35:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj863 datafile 2023-12-02T01:35:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj864 datafile 2023-12-02T01:35:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj865 datafile 2023-12-02T01:35:24.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:24.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj866 datafile 2023-12-02T01:35:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj867 datafile 2023-12-02T01:35:25.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:25.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj868 datafile 2023-12-02T01:35:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj869 datafile 2023-12-02T01:35:25.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:25.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj870 datafile 2023-12-02T01:35:25.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:25.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj871 datafile 2023-12-02T01:35:25.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:25.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj872 datafile 2023-12-02T01:35:25.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:25.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj873 datafile 2023-12-02T01:35:26.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:26.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj874 datafile 2023-12-02T01:35:26.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:26.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj875 datafile 2023-12-02T01:35:26.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:26.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj876 datafile 2023-12-02T01:35:26.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:26.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj877 datafile 2023-12-02T01:35:26.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:26.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj878 datafile 2023-12-02T01:35:26.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:26.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj879 datafile 2023-12-02T01:35:27.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:27.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj880 datafile 2023-12-02T01:35:27.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:27.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj881 datafile 2023-12-02T01:35:27.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:27.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj882 datafile 2023-12-02T01:35:27.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:27.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj883 datafile 2023-12-02T01:35:27.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:27.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj884 datafile 2023-12-02T01:35:27.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:27.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj885 datafile 2023-12-02T01:35:27.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:27.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj886 datafile 2023-12-02T01:35:28.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:28.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj887 datafile 2023-12-02T01:35:28.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:28.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj888 datafile 2023-12-02T01:35:28.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:28.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj889 datafile 2023-12-02T01:35:28.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:28.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj890 datafile 2023-12-02T01:35:28.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:28.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj891 datafile 2023-12-02T01:35:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj892 datafile 2023-12-02T01:35:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj893 datafile 2023-12-02T01:35:29.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:29.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj894 datafile 2023-12-02T01:35:29.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:29.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj895 datafile 2023-12-02T01:35:29.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:29.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj896 datafile 2023-12-02T01:35:29.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:29.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj897 datafile 2023-12-02T01:35:29.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:29.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj898 datafile 2023-12-02T01:35:29.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:29.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj899 datafile 2023-12-02T01:35:30.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:30.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj900 datafile 2023-12-02T01:35:30.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:30.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj901 datafile 2023-12-02T01:35:30.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:30.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj902 datafile 2023-12-02T01:35:30.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:30.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj903 datafile 2023-12-02T01:35:30.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:30.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj904 datafile 2023-12-02T01:35:30.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:30.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj905 datafile 2023-12-02T01:35:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj906 datafile 2023-12-02T01:35:31.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:31.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj907 datafile 2023-12-02T01:35:31.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:31.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj908 datafile 2023-12-02T01:35:31.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:31.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj909 datafile 2023-12-02T01:35:31.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:31.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj910 datafile 2023-12-02T01:35:31.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:31.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj911 datafile 2023-12-02T01:35:31.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:31.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj912 datafile 2023-12-02T01:35:31.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:31.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj913 datafile 2023-12-02T01:35:32.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:32.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj914 datafile 2023-12-02T01:35:32.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:32.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj915 datafile 2023-12-02T01:35:32.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:32.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj916 datafile 2023-12-02T01:35:32.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:32.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj917 datafile 2023-12-02T01:35:32.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:32.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj918 datafile 2023-12-02T01:35:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:32.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj919 datafile 2023-12-02T01:35:32.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:32.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj920 datafile 2023-12-02T01:35:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj921 datafile 2023-12-02T01:35:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj922 datafile 2023-12-02T01:35:33.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:33.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj923 datafile 2023-12-02T01:35:33.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:33.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj924 datafile 2023-12-02T01:35:33.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:33.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj925 datafile 2023-12-02T01:35:33.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:33.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj926 datafile 2023-12-02T01:35:33.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:33.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj927 datafile 2023-12-02T01:35:34.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:34.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj928 datafile 2023-12-02T01:35:34.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:34.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj929 datafile 2023-12-02T01:35:34.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:34.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj930 datafile 2023-12-02T01:35:34.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:34.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj931 datafile 2023-12-02T01:35:34.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:34.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj932 datafile 2023-12-02T01:35:34.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:34.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj933 datafile 2023-12-02T01:35:34.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:34.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj934 datafile 2023-12-02T01:35:35.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:35.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj935 datafile 2023-12-02T01:35:35.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:35.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj936 datafile 2023-12-02T01:35:35.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:35.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj937 datafile 2023-12-02T01:35:35.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:35.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj938 datafile 2023-12-02T01:35:35.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:35.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj939 datafile 2023-12-02T01:35:35.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:35.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj940 datafile 2023-12-02T01:35:35.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:35.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj941 datafile 2023-12-02T01:35:36.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:36.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj942 datafile 2023-12-02T01:35:36.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:36.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj943 datafile 2023-12-02T01:35:36.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:36.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj944 datafile 2023-12-02T01:35:36.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:36.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj945 datafile 2023-12-02T01:35:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj946 datafile 2023-12-02T01:35:36.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:36.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj947 datafile 2023-12-02T01:35:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj948 datafile 2023-12-02T01:35:37.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:37.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj949 datafile 2023-12-02T01:35:37.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:37.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj950 datafile 2023-12-02T01:35:37.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:37.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj951 datafile 2023-12-02T01:35:37.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:37.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj952 datafile 2023-12-02T01:35:37.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:37.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj953 datafile 2023-12-02T01:35:37.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:37.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj954 datafile 2023-12-02T01:35:38.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:38.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj955 datafile 2023-12-02T01:35:38.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:38.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj956 datafile 2023-12-02T01:35:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj957 datafile 2023-12-02T01:35:38.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:38.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj958 datafile 2023-12-02T01:35:38.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:38.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj959 datafile 2023-12-02T01:35:38.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:38.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj960 datafile 2023-12-02T01:35:38.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:38.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj961 datafile 2023-12-02T01:35:39.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj962 datafile 2023-12-02T01:35:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj963 datafile 2023-12-02T01:35:39.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:39.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj964 datafile 2023-12-02T01:35:39.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj965 datafile 2023-12-02T01:35:39.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:39.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj966 datafile 2023-12-02T01:35:39.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:39.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj967 datafile 2023-12-02T01:35:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj968 datafile 2023-12-02T01:35:40.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:40.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj969 datafile 2023-12-02T01:35:40.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:40.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj970 datafile 2023-12-02T01:35:40.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:40.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj971 datafile 2023-12-02T01:35:40.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:40.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj972 datafile 2023-12-02T01:35:40.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:40.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj973 datafile 2023-12-02T01:35:40.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:40.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj974 datafile 2023-12-02T01:35:40.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:40.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj975 datafile 2023-12-02T01:35:41.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:41.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj976 datafile 2023-12-02T01:35:41.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:41.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj977 datafile 2023-12-02T01:35:41.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:41.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj978 datafile 2023-12-02T01:35:41.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:41.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj979 datafile 2023-12-02T01:35:41.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:41.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj980 datafile 2023-12-02T01:35:41.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:41.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj981 datafile 2023-12-02T01:35:41.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:41.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj982 datafile 2023-12-02T01:35:42.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:42.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj983 datafile 2023-12-02T01:35:42.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:42.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj984 datafile 2023-12-02T01:35:42.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:42.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj985 datafile 2023-12-02T01:35:42.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:42.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj986 datafile 2023-12-02T01:35:43.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:43.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj987 datafile 2023-12-02T01:35:43.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:43.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj988 datafile 2023-12-02T01:35:43.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:43.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj989 datafile 2023-12-02T01:35:43.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:43.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj990 datafile 2023-12-02T01:35:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:43.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj991 datafile 2023-12-02T01:35:43.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:43.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj992 datafile 2023-12-02T01:35:43.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:43.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj993 datafile 2023-12-02T01:35:44.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:44.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj994 datafile 2023-12-02T01:35:44.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:44.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj995 datafile 2023-12-02T01:35:44.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:44.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj996 datafile 2023-12-02T01:35:44.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:44.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj997 datafile 2023-12-02T01:35:44.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:44.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj998 datafile 2023-12-02T01:35:44.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:44.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj999 datafile 2023-12-02T01:35:44.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2023-12-02T01:35:44.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj1000 datafile 2023-12-02T01:35:45.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:81: TEST_recover_unexpected: rm datafile 2023-12-02T01:35:45.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:83: TEST_recover_unexpected: ceph osd set noscrub 2023-12-02T01:35:45.468 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2023-12-02T01:35:45.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: ceph osd set nodeep-scrub 2023-12-02T01:35:45.988 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2023-12-02T01:35:46.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json-pretty 2023-12-02T01:35:46.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[].pgid' 2023-12-02T01:35:46.268 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:35:46.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:46.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.9"' 2023-12-02T01:35:46.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.9 2023-12-02T01:35:46.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.9 scrub 2023-12-02T01:35:46.386 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:46.386 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:46.386 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:46.386 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:46.386 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:46.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:46.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.b"' 2023-12-02T01:35:46.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.b 2023-12-02T01:35:46.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.b scrub 2023-12-02T01:35:46.501 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:46.501 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:46.501 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:46.501 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:46.501 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:46.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:46.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.8"' 2023-12-02T01:35:46.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.8 2023-12-02T01:35:46.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.8 scrub 2023-12-02T01:35:46.614 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:46.615 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:46.615 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:46.616 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:46.616 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:46.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:46.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.8"' 2023-12-02T01:35:46.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.8 2023-12-02T01:35:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.8 scrub 2023-12-02T01:35:46.728 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:46.728 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:46.729 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:46.729 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:46.729 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:46.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:46.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.a"' 2023-12-02T01:35:46.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.a 2023-12-02T01:35:46.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.a scrub 2023-12-02T01:35:46.842 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:46.842 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:46.842 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:46.842 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:46.842 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:46.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:46.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.9"' 2023-12-02T01:35:46.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.9 2023-12-02T01:35:46.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.9 scrub 2023-12-02T01:35:46.957 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:46.957 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:46.957 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:46.957 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:46.958 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:46.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:46.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.b"' 2023-12-02T01:35:46.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.b 2023-12-02T01:35:46.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.b scrub 2023-12-02T01:35:47.086 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.087 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.087 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.087 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.087 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.9"' 2023-12-02T01:35:47.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.9 2023-12-02T01:35:47.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.9 scrub 2023-12-02T01:35:47.201 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.201 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.201 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.201 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.202 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.a"' 2023-12-02T01:35:47.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.a 2023-12-02T01:35:47.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.a scrub 2023-12-02T01:35:47.316 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.316 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.316 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.316 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.316 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.a"' 2023-12-02T01:35:47.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.a 2023-12-02T01:35:47.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.a scrub 2023-12-02T01:35:47.428 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.429 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.429 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.429 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.429 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.8"' 2023-12-02T01:35:47.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.8 2023-12-02T01:35:47.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.8 scrub 2023-12-02T01:35:47.542 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.542 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.542 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.542 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.542 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.b"' 2023-12-02T01:35:47.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.b 2023-12-02T01:35:47.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.b scrub 2023-12-02T01:35:47.655 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.655 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.655 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.655 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.656 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.5"' 2023-12-02T01:35:47.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.5 2023-12-02T01:35:47.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.5 scrub 2023-12-02T01:35:47.769 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.770 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.770 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.770 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.770 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.7"' 2023-12-02T01:35:47.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.7 2023-12-02T01:35:47.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.7 scrub 2023-12-02T01:35:47.883 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:47.884 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:47.884 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:47.884 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:47.884 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:47.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:47.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.4"' 2023-12-02T01:35:47.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.4 2023-12-02T01:35:47.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.4 scrub 2023-12-02T01:35:48.003 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.003 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.003 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.003 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.003 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.4"' 2023-12-02T01:35:48.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.4 2023-12-02T01:35:48.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.4 scrub 2023-12-02T01:35:48.117 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.117 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.118 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.118 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.118 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.6"' 2023-12-02T01:35:48.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.6 2023-12-02T01:35:48.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.6 scrub 2023-12-02T01:35:48.232 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.232 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.233 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.233 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.233 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.5"' 2023-12-02T01:35:48.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.5 2023-12-02T01:35:48.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.5 scrub 2023-12-02T01:35:48.346 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.346 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.346 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.346 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.346 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.7"' 2023-12-02T01:35:48.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.7 2023-12-02T01:35:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.7 scrub 2023-12-02T01:35:48.461 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.462 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.462 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.462 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.462 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.5"' 2023-12-02T01:35:48.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.5 2023-12-02T01:35:48.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.5 scrub 2023-12-02T01:35:48.576 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.576 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.576 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.576 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.576 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.6"' 2023-12-02T01:35:48.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.6 2023-12-02T01:35:48.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.6 scrub 2023-12-02T01:35:48.700 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.700 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.700 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.700 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.700 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.6"' 2023-12-02T01:35:48.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.6 2023-12-02T01:35:48.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.6 scrub 2023-12-02T01:35:48.815 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.815 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.815 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.815 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.815 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.4"' 2023-12-02T01:35:48.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.4 2023-12-02T01:35:48.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.4 scrub 2023-12-02T01:35:48.929 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:48.929 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:48.929 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:48.929 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:48.929 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:48.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:48.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.7"' 2023-12-02T01:35:48.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.7 2023-12-02T01:35:48.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.7 scrub 2023-12-02T01:35:49.043 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.043 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.043 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.044 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.044 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.1"' 2023-12-02T01:35:49.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.1 2023-12-02T01:35:49.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.1 scrub 2023-12-02T01:35:49.157 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.157 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.157 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.157 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.157 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.3"' 2023-12-02T01:35:49.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.3 2023-12-02T01:35:49.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.3 scrub 2023-12-02T01:35:49.272 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.272 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.272 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.272 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.272 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.0"' 2023-12-02T01:35:49.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.0 2023-12-02T01:35:49.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.0 scrub 2023-12-02T01:35:49.385 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.385 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.385 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.385 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.385 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.0"' 2023-12-02T01:35:49.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.0 2023-12-02T01:35:49.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.0 scrub 2023-12-02T01:35:49.500 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.500 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.500 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.500 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.500 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.2"' 2023-12-02T01:35:49.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.2 2023-12-02T01:35:49.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.2 scrub 2023-12-02T01:35:49.616 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.616 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.616 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.616 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.616 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.1"' 2023-12-02T01:35:49.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.1 2023-12-02T01:35:49.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.1 scrub 2023-12-02T01:35:49.730 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.730 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.730 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.730 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.731 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.e"' 2023-12-02T01:35:49.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.e 2023-12-02T01:35:49.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.e scrub 2023-12-02T01:35:49.845 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.845 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.845 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.845 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.845 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:49.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.f"' 2023-12-02T01:35:49.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.f 2023-12-02T01:35:49.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.f scrub 2023-12-02T01:35:49.990 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:49.990 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:49.990 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:49.990 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:49.991 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:49.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.c"' 2023-12-02T01:35:50.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.c 2023-12-02T01:35:50.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.c scrub 2023-12-02T01:35:50.193 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:50.193 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:50.193 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:50.194 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:50.194 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:50.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.3"' 2023-12-02T01:35:50.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.3 2023-12-02T01:35:50.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.3 scrub 2023-12-02T01:35:50.312 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:50.312 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:50.312 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:50.313 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:50.313 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:50.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.1"' 2023-12-02T01:35:50.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.1 2023-12-02T01:35:50.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.1 scrub 2023-12-02T01:35:50.425 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:50.426 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:50.426 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:50.426 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:50.426 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:50.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.2"' 2023-12-02T01:35:50.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.2 2023-12-02T01:35:50.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.2 scrub 2023-12-02T01:35:50.539 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:50.539 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:50.540 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:50.540 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:50.540 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:50.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.2"' 2023-12-02T01:35:50.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.2 2023-12-02T01:35:50.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.2 scrub 2023-12-02T01:35:50.653 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:50.653 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:50.653 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:50.653 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:50.653 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.0"' 2023-12-02T01:35:50.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.0 2023-12-02T01:35:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.0 scrub 2023-12-02T01:35:50.767 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:50.767 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:50.767 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:50.767 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:50.767 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:50.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.3"' 2023-12-02T01:35:50.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.3 2023-12-02T01:35:50.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.3 scrub 2023-12-02T01:35:50.902 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:50.903 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:50.903 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:50.903 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:50.903 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.f"' 2023-12-02T01:35:50.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.f 2023-12-02T01:35:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.f scrub 2023-12-02T01:35:51.016 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.017 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.017 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.017 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.017 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.e"' 2023-12-02T01:35:51.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.e 2023-12-02T01:35:51.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.e scrub 2023-12-02T01:35:51.153 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.154 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.154 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.154 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.154 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.d"' 2023-12-02T01:35:51.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.d 2023-12-02T01:35:51.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.d scrub 2023-12-02T01:35:51.268 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.268 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.268 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.268 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.268 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.c"' 2023-12-02T01:35:51.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.c 2023-12-02T01:35:51.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.c scrub 2023-12-02T01:35:51.381 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.382 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.382 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.382 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.382 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.d"' 2023-12-02T01:35:51.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.d 2023-12-02T01:35:51.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.d scrub 2023-12-02T01:35:51.496 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.496 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.496 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.497 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.497 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.e"' 2023-12-02T01:35:51.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.e 2023-12-02T01:35:51.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.e scrub 2023-12-02T01:35:51.610 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.610 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.610 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.610 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.610 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="3.d"' 2023-12-02T01:35:51.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=3.d 2023-12-02T01:35:51.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 3.d scrub 2023-12-02T01:35:51.722 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.722 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.723 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.723 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.723 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="1.f"' 2023-12-02T01:35:51.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=1.f 2023-12-02T01:35:51.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 1.f scrub 2023-12-02T01:35:51.837 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.837 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.837 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.837 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.837 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2023-12-02T01:35:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: eval 'pg="2.c"' 2023-12-02T01:35:51.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: pg=2.c 2023-12-02T01:35:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: ceph tell 2.c scrub 2023-12-02T01:35:51.952 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:51.952 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2023-12-02T01:35:51.952 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2023-12-02T01:35:51.952 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2023-12-02T01:35:51.952 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:51.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:92: TEST_recover_unexpected: ceph pg dump pgs 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:3.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:05.617103+0000 42'61 44:80 [4,2,5] 4 [4,2,5] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:1.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2023-12-02T01:28:01.469773+0000 42'65 43:92 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:2.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2023-12-02T01:35:46.877679+0000 42'66 44:93 [0,2,1] 0 [0,2,1] 0 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 scrubbing for 0s 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:3.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2023-12-02T01:28:05.566895+0000 42'66 44:85 [4,1,2] 4 [4,1,2] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:1.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:01.471860+0000 42'61 43:88 [1,2,4] 1 [1,2,4] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:2.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:03.719136+0000 42'61 44:84 [4,1,0] 4 [4,1,0] 4 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:3.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2023-12-02T01:28:05.566847+0000 42'65 43:84 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:1.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:01.295717+0000 42'61 44:88 [1,4,5] 1 [1,4,5] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:2.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:03.719539+0000 42'61 44:84 [1,5,0] 1 [1,5,0] 1 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.235 INFO:tasks.workunit.client.0.smithi162.stdout:3.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:05.617211+0000 42'61 43:80 [3,1,5] 3 [3,1,5] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2023-12-02T01:35:49.191330+0000 42'66 44:95 [5,4,3] 5 [5,4,3] 5 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 scrubbing for 0s 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:2.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2023-12-02T01:28:03.560905+0000 42'65 43:88 [3,2,5] 3 [3,2,5] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:3.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2023-12-02T01:28:05.617012+0000 42'73 43:92 [5,3,4] 5 [5,3,4] 5 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:1.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2023-12-02T01:28:01.295912+0000 42'53 44:80 [1,3,0] 1 [1,3,0] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:2.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2023-12-02T01:28:03.719160+0000 42'51 44:74 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:3.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2023-12-02T01:28:05.566975+0000 42'51 44:70 [1,2,4] 1 [1,2,4] 1 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:1.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2023-12-02T01:28:01.470658+0000 42'64 43:91 [3,4,2] 3 [3,4,2] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:2.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2023-12-02T01:28:03.630991+0000 42'73 43:96 [3,4,0] 3 [3,4,0] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:3.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2023-12-02T01:28:05.568621+0000 42'53 43:71 [4,3,0] 4 [4,3,0] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T08:05:13.445803+0000 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:1.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2023-12-02T01:28:01.469805+0000 42'73 43:99 [4,2,5] 4 [4,2,5] 4 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:56:37.929328+0000 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:2.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2023-12-02T01:28:03.719031+0000 42'64 44:87 [1,0,4] 1 [1,0,4] 1 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:3.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2023-12-02T01:28:05.567223+0000 42'64 43:82 [0,1,4] 0 [0,1,4] 0 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T07:58:29.323251+0000 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:1.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean+scrubbing 2023-12-02T01:35:49.409019+0000 42'51 44:79 [3,0,1] 3 [3,0,1] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:2.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2023-12-02T01:28:03.719017+0000 42'53 44:76 [3,0,2] 3 [3,0,2] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.236 INFO:tasks.workunit.client.0.smithi162.stdout:3.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:05.617530+0000 42'67 43:85 [4,3,5] 4 [4,3,5] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T03:54:41.689650+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2023-12-02T01:28:01.469417+0000 42'54 43:80 [4,1,2] 4 [4,1,2] 4 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T11:41:36.611971+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2023-12-02T01:28:03.719648+0000 42'72 44:95 [3,5,1] 3 [3,5,1] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:3.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2023-12-02T01:28:05.568635+0000 42'72 43:90 [1,2,4] 1 [1,2,4] 1 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T12:37:39.601002+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2023-12-02T01:28:01.471483+0000 42'55 43:81 [0,4,1] 0 [0,4,1] 0 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T07:32:50.664760+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:2.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:03.719888+0000 42'67 44:90 [2,5,0] 2 [2,5,0] 2 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:3.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2023-12-02T01:28:05.566794+0000 42'69 44:88 [2,4,1] 2 [2,4,1] 2 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:2.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2023-12-02T01:28:03.560803+0000 42'62 43:84 [0,5,4] 0 [0,5,4] 0 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:47:22.923121+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:1.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2023-12-02T01:28:01.472429+0000 42'60 44:87 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:3.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2023-12-02T01:28:05.617117+0000 42'54 43:72 [4,0,5] 4 [4,0,5] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:30:22.892015+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:01.472630+0000 42'67 43:91 [5,0,4] 5 [5,0,4] 5 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:11:23.167269+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:2.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2023-12-02T01:28:03.722272+0000 42'55 43:75 [5,1,0] 5 [5,1,0] 5 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:09:33.906781+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:3.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2023-12-02T01:28:05.617095+0000 42'55 44:74 [3,5,0] 3 [3,5,0] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2023-12-02T01:28:01.469522+0000 42'72 43:98 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T08:52:11.245257+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:2.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2023-12-02T01:28:03.722727+0000 42'54 43:74 [5,4,1] 5 [5,4,1] 5 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:21:54.540463+0000 0 0 2023-12-02T01:35:52.237 INFO:tasks.workunit.client.0.smithi162.stdout:3.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2023-12-02T01:28:05.567671+0000 42'62 43:80 [0,1,4] 0 [0,1,4] 0 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T03:18:11.841579+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:2.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2023-12-02T01:28:03.719026+0000 42'69 43:91 [2,3,5] 2 [2,3,5] 2 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:26:32.663678+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:1.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:01.295830+0000 42'67 43:93 [2,4,1] 2 [2,4,1] 2 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T07:23:51.137531+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:3.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2023-12-02T01:28:05.618077+0000 42'60 43:78 [5,3,1] 5 [5,3,1] 5 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:27:29.323210+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:2.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:03.722527+0000 42'67 43:87 [5,1,3] 5 [5,1,3] 5 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:04:32.844324+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:1.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2023-12-02T01:28:01.470864+0000 42'69 43:95 [1,4,2] 1 [1,4,2] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T05:59:39.269758+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:3.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:05.568209+0000 42'67 43:85 [3,1,0] 3 [3,1,0] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T04:09:37.029755+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:1.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2023-12-02T01:28:01.471776+0000 42'62 43:88 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T11:04:09.527377+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:2.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2023-12-02T01:28:03.720625+0000 42'60 43:82 [2,0,5] 2 [2,0,5] 2 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:00:43.473530+0000 0 0 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:35:52.238 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:94: TEST_recover_unexpected: jq .osd_max_scrubs 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:94: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:52.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:94: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:52.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:94: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:52.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:94: TEST_recover_unexpected: max=4 2023-12-02T01:35:52.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:95: TEST_recover_unexpected: '[' 4 '!=' 4 ']' 2023-12-02T01:35:52.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:100: TEST_recover_unexpected: ceph osd unset noscrub 2023-12-02T01:35:52.701 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is unset 2023-12-02T01:35:52.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:102: TEST_recover_unexpected: ok=false 2023-12-02T01:35:52.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: seq 0 300 2023-12-02T01:35:52.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2023-12-02T01:35:52.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2023-12-02T01:35:52.976 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-12-02T01:35:52.976 INFO:tasks.workunit.client.0.smithi162.stdout:3.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:05.617103+0000 42'61 44:80 [4,2,5] 4 [4,2,5] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.976 INFO:tasks.workunit.client.0.smithi162.stdout:1.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2023-12-02T01:28:01.469773+0000 42'65 43:92 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.976 INFO:tasks.workunit.client.0.smithi162.stdout:2.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2023-12-02T01:35:46.877679+0000 42'66 44:94 [0,2,1] 0 [0,2,1] 0 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 scrubbing for 3s 50 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:3.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2023-12-02T01:28:05.566895+0000 42'66 44:85 [4,1,2] 4 [4,1,2] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:1.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:01.471860+0000 42'61 43:88 [1,2,4] 1 [1,2,4] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:2.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:03.719136+0000 42'61 44:84 [4,1,0] 4 [4,1,0] 4 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:3.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2023-12-02T01:28:05.566847+0000 42'65 43:84 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:1.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:01.295717+0000 42'61 44:88 [1,4,5] 1 [1,4,5] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:2.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:03.719539+0000 42'61 44:84 [1,5,0] 1 [1,5,0] 1 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:3.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2023-12-02T01:28:05.617211+0000 42'61 43:80 [3,1,5] 3 [3,1,5] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2023-12-02T01:35:49.191330+0000 42'66 44:95 [5,4,3] 5 [5,4,3] 5 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 scrubbing for 0s 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:2.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2023-12-02T01:28:03.560905+0000 42'65 43:88 [3,2,5] 3 [3,2,5] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:3.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2023-12-02T01:28:05.617012+0000 42'73 43:92 [5,3,4] 5 [5,3,4] 5 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:1.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2023-12-02T01:28:01.295912+0000 42'53 44:80 [1,3,0] 1 [1,3,0] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:2.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2023-12-02T01:28:03.719160+0000 42'51 44:74 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:3.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2023-12-02T01:28:05.566975+0000 42'51 44:70 [1,2,4] 1 [1,2,4] 1 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.977 INFO:tasks.workunit.client.0.smithi162.stdout:1.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2023-12-02T01:28:01.470658+0000 42'64 43:91 [3,4,2] 3 [3,4,2] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:2.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2023-12-02T01:28:03.630991+0000 42'73 43:96 [3,4,0] 3 [3,4,0] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:3.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2023-12-02T01:28:05.568621+0000 42'53 43:71 [4,3,0] 4 [4,3,0] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T08:05:13.445803+0000 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:1.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2023-12-02T01:28:01.469805+0000 42'73 43:99 [4,2,5] 4 [4,2,5] 4 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T01:56:37.929328+0000 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:2.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2023-12-02T01:28:03.719031+0000 42'64 44:87 [1,0,4] 1 [1,0,4] 1 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:3.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean+scrubbing 2023-12-02T01:35:48.851517+0000 42'64 44:88 [0,1,4] 0 [0,1,4] 0 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 scrubbing for 3s 50 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:1.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean+scrubbing 2023-12-02T01:35:49.409019+0000 42'51 44:79 [3,0,1] 3 [3,0,1] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:2.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2023-12-02T01:28:03.719017+0000 42'53 44:76 [3,0,2] 3 [3,0,2] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:3.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:05.617530+0000 42'67 43:85 [4,3,5] 4 [4,3,5] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T03:54:41.689650+0000 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2023-12-02T01:28:01.469417+0000 42'54 43:80 [4,1,2] 4 [4,1,2] 4 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T11:41:36.611971+0000 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2023-12-02T01:28:03.719648+0000 42'72 44:95 [3,5,1] 3 [3,5,1] 3 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:3.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2023-12-02T01:28:05.568635+0000 42'72 43:90 [1,2,4] 1 [1,2,4] 1 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T12:37:39.601002+0000 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2023-12-02T01:28:01.471483+0000 42'55 44:82 [0,4,1] 0 [0,4,1] 0 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:2.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:03.719888+0000 42'67 44:90 [2,5,0] 2 [2,5,0] 2 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:3.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2023-12-02T01:28:05.566794+0000 42'69 44:88 [2,4,1] 2 [2,4,1] 2 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.978 INFO:tasks.workunit.client.0.smithi162.stdout:2.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2023-12-02T01:28:03.560803+0000 42'62 44:85 [0,5,4] 0 [0,5,4] 0 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:1.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2023-12-02T01:28:01.472429+0000 42'60 44:87 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:3.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2023-12-02T01:28:05.617117+0000 42'54 43:72 [4,0,5] 4 [4,0,5] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:30:22.892015+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:01.472630+0000 42'67 43:91 [5,0,4] 5 [5,0,4] 5 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:11:23.167269+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:2.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2023-12-02T01:28:03.722272+0000 42'55 43:75 [5,1,0] 5 [5,1,0] 5 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:09:33.906781+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:3.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2023-12-02T01:28:05.617095+0000 42'55 44:74 [3,5,0] 3 [3,5,0] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2023-12-02T01:28:01.469522+0000 42'72 43:98 [1,0,2] 1 [1,0,2] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T08:52:11.245257+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:2.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2023-12-02T01:28:03.722727+0000 42'54 43:74 [5,4,1] 5 [5,4,1] 5 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:21:54.540463+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:3.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2023-12-02T01:28:05.567671+0000 42'62 44:81 [0,1,4] 0 [0,1,4] 0 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:2.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2023-12-02T01:28:03.719026+0000 42'69 43:91 [2,3,5] 2 [2,3,5] 2 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:26:32.663678+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:1.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:01.295830+0000 42'67 43:93 [2,4,1] 2 [2,4,1] 2 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T07:23:51.137531+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:3.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2023-12-02T01:28:05.618077+0000 42'60 43:78 [5,3,1] 5 [5,3,1] 5 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T13:27:29.323210+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:2.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:03.722527+0000 42'67 43:87 [5,1,3] 5 [5,1,3] 5 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:04:32.844324+0000 0 0 2023-12-02T01:35:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:1.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2023-12-02T01:28:01.470864+0000 42'69 43:95 [1,4,2] 1 [1,4,2] 1 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T05:59:39.269758+0000 0 0 2023-12-02T01:35:52.981 INFO:tasks.workunit.client.0.smithi162.stdout:3.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2023-12-02T01:28:05.568209+0000 42'67 43:85 [3,1,0] 3 [3,1,0] 3 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 periodic scrub scheduled @ 2023-12-03T04:09:37.029755+0000 0 0 2023-12-02T01:35:52.981 INFO:tasks.workunit.client.0.smithi162.stdout:1.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2023-12-02T01:28:01.471776+0000 42'62 43:88 [3,0,4] 3 [3,0,4] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 periodic scrub scheduled @ 2023-12-03T11:04:09.527377+0000 0 0 2023-12-02T01:35:52.981 INFO:tasks.workunit.client.0.smithi162.stdout:2.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2023-12-02T01:28:03.720625+0000 42'60 43:82 [2,0,5] 2 [2,0,5] 2 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 periodic scrub scheduled @ 2023-12-03T06:00:43.473530+0000 0 0 2023-12-02T01:35:52.981 INFO:tasks.workunit.client.0.smithi162.stdout: 2023-12-02T01:35:52.981 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-12-02T01:35:52.981 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:35:52.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2023-12-02T01:35:52.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep +scrubbing 2023-12-02T01:35:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:35:53.289 INFO:tasks.workunit.client.0.smithi162.stdout:2.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2023-12-02T01:35:46.877679+0000 42'66 44:94 [0,2,1] 0 [0,2,1] 0 0'0 2023-12-02T01:28:01.863174+0000 0'0 2023-12-02T01:28:01.863174+0000 0 0 scrubbing for 3s 50 0 2023-12-02T01:35:53.289 INFO:tasks.workunit.client.0.smithi162.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2023-12-02T01:35:49.191330+0000 42'66 44:95 [5,4,3] 5 [5,4,3] 5 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 scrubbing for 0s 0 0 2023-12-02T01:35:53.289 INFO:tasks.workunit.client.0.smithi162.stdout:3.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean+scrubbing 2023-12-02T01:35:48.851517+0000 42'64 44:88 [0,1,4] 0 [0,1,4] 0 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 scrubbing for 3s 50 0 2023-12-02T01:35:53.289 INFO:tasks.workunit.client.0.smithi162.stdout:1.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean+scrubbing 2023-12-02T01:35:49.409019+0000 42'51 44:79 [3,0,1] 3 [3,0,1] 3 0'0 2023-12-02T01:27:59.916779+0000 0'0 2023-12-02T01:27:59.916779+0000 0 0 queued for scrub 0 0 2023-12-02T01:35:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: ok=true 2023-12-02T01:35:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:108: TEST_recover_unexpected: break 2023-12-02T01:35:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:112: TEST_recover_unexpected: test true = false 2023-12-02T01:35:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:117: TEST_recover_unexpected: local total=0 2023-12-02T01:35:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:118: TEST_recover_unexpected: local zerocount=0 2023-12-02T01:35:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:119: TEST_recover_unexpected: local maxzerocount=3 2023-12-02T01:35:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:35:53.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:35:53.291 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:35:53.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:35:53.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:53.372 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:53.372 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:53.372 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:35:53.372 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:53.372 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:53.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:53.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:53.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:53.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:53.381 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.381 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.381 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:53.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:53.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:35:53.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:35:53.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:35:53.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:35:53.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:53.550 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:53.551 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:53.551 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:35:53.551 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:53.551 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:53.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:53.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:35:53.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:35:53.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2023-12-02T01:35:53.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:35:53.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:53.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:53.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:53.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:53.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:53.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:35:53.729 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:53.729 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:53.730 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:35:53.730 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:53.730 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:53.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:35:53.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:35:53.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:35:53.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 1 2023-12-02T01:35:53.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:35:53.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:53.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:35:53.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:35:53.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:35:53.828 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.828 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.828 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:35:53.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:35:53.909 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:53.909 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:35:53.909 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:53.909 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:53.909 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:53.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:53.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:35:53.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:35:53.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:35:53.918 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:53.918 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:53.918 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:53.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:35:53.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:53.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:35:54.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:35:54.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:35:54.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2023-12-02T01:35:54.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:35:54.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:54.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:35:54.086 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:54.086 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:54.086 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:35:54.086 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:54.087 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:54.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:35:54.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:35:54.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:35:54.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2023-12-02T01:35:54.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:54.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:35:54.264 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:54.264 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:35:54.264 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:35:54.264 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:54.264 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:54.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:35:54.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:35:54.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:35:54.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2023-12-02T01:35:54.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2023-12-02T01:35:54.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2023-12-02T01:35:54.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:35:54.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 0 + 18 2023-12-02T01:35:54.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=18 2023-12-02T01:35:54.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 18 -gt 0 ']' 2023-12-02T01:35:54.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:35:54.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:35:55.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:35:55.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:35:55.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:35:55.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:35:55.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:55.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:55.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:55.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:55.369 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.369 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.370 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:55.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:55.456 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:55.457 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:35:55.457 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:35:55.457 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:55.457 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:55.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:55.466 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:55.466 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:55.466 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:55.466 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.466 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.467 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:55.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:55.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:35:55.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:35:55.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:35:55.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:35:55.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:55.636 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:55.636 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:35:55.637 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:55.637 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:55.637 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:55.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:55.645 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:55.645 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:55.645 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:55.645 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.646 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.646 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.646 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:55.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:55.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:35:55.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:35:55.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2023-12-02T01:35:55.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:35:55.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:35:55.813 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:55.813 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:35:55.813 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:35:55.813 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:55.813 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:35:55.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:35:55.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:35:55.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2023-12-02T01:35:55.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:35:55.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:55.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:35:55.990 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:55.990 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:35:55.990 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:55.990 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:55.990 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:55.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:35:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:35:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:35:55.999 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:55.999 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:55.999 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:55.999 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:35:55.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:55.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:35:56.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:35:56.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:35:56.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 1 2023-12-02T01:35:56.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:35:56.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:56.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:35:56.168 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:56.168 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:35:56.168 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:56.168 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:56.168 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:56.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:56.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:35:56.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:35:56.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:35:56.177 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:56.178 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:56.178 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:56.178 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:35:56.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:56.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:35:56.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:35:56.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:35:56.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2023-12-02T01:35:56.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:35:56.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:56.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:35:56.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:35:56.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:35:56.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:56.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:56.267 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:56.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:35:56.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:56.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:35:56.345 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:56.345 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:56.345 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:35:56.346 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:56.346 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:35:56.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:56.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:35:56.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:35:56.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:35:56.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2023-12-02T01:35:56.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:35:56.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 10 = 0 ']' 2023-12-02T01:35:56.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:35:56.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 18 + 10 2023-12-02T01:35:56.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=28 2023-12-02T01:35:56.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 28 -gt 0 ']' 2023-12-02T01:35:56.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:35:56.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:35:57.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:35:57.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:35:57.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:35:57.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:35:57.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:57.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:57.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:57.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:57.450 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:57.450 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:57.450 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:57.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:57.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:57.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:57.537 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:57.537 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:57.537 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:35:57.537 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:57.538 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:57.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:57.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:57.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:35:57.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:35:57.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:35:57.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:35:57.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:57.717 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:57.717 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:35:57.717 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:35:57.717 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:57.717 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:57.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:57.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:35:57.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:35:57.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2023-12-02T01:35:57.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:35:57.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:57.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:57.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:57.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:57.814 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:57.814 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:57.814 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:57.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:35:57.892 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:57.892 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:57.892 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:57.892 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:57.892 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:57.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:57.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:35:57.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:35:57.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:35:57.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2023-12-02T01:35:57.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:35:57.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:57.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:35:57.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:35:57.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:35:57.988 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:57.988 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:57.988 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:57.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:35:57.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:57.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:35:58.068 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:58.068 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:35:58.068 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:35:58.068 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:58.068 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:35:58.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:58.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:35:58.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:35:58.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:35:58.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 2 2023-12-02T01:35:58.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:35:58.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:58.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:35:58.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:35:58.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:35:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:58.166 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:58.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:35:58.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:58.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:35:58.243 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:58.243 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:35:58.243 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:58.243 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:58.243 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:58.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:58.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:35:58.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:35:58.252 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:35:58.252 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:58.252 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:58.252 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:58.252 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:35:58.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:58.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:35:58.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:35:58.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:35:58.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 1 2023-12-02T01:35:58.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2023-12-02T01:35:58.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:58.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:35:58.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:35:58.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:35:58.340 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:58.341 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:58.341 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:58.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:35:58.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:58.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:35:58.418 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:58.418 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:58.418 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:58.418 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:58.418 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:58.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:58.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:58.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:35:58.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:35:58.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:35:58.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 2 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 28 + 15 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=43 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 43 -gt 0 ']' 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:35:58.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:35:59.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:35:59.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:35:59.519 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:35:59.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:35:59.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:59.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:59.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:59.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:59.522 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:59.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:59.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:59.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:59.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:59.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:59.611 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:59.611 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:59.611 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:35:59.611 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:59.611 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:59.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:59.620 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:35:59.620 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:35:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:35:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:35:59.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:59.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:35:59.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:35:59.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:35:59.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:35:59.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:35:59.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:59.789 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:59.789 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:35:59.790 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:35:59.790 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:59.790 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:59.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:59.798 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:35:59.798 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:35:59.798 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:35:59.798 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:59.798 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:59.799 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:59.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:35:59.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:59.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:35:59.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:35:59.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:35:59.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2023-12-02T01:35:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:35:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:59.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:59.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:35:59.967 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:35:59.967 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:35:59.968 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:35:59.968 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:35:59.968 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:35:59.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:35:59.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:00.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2023-12-02T01:36:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:00.146 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:00.146 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:00.146 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:00.147 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:00.147 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:00.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:00.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:00.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2023-12-02T01:36:00.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:00.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:00.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:00.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:00.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:00.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:00.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:00.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:00.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:00.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:00.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:00.324 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:00.324 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:00.324 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:00.324 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:00.324 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:00.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:00.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:00.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:00.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:00.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:00.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2023-12-02T01:36:00.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2023-12-02T01:36:00.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:00.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:00.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:00.500 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:00.501 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:00.501 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:00.501 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:00.501 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:00.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:00.509 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:00.509 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:00.509 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:00.509 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:00.509 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:00.510 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:00.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:00.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:00.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:00.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:00.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:00.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 3 2023-12-02T01:36:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2023-12-02T01:36:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2023-12-02T01:36:00.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:00.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 43 + 17 2023-12-02T01:36:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=60 2023-12-02T01:36:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 60 -gt 0 ']' 2023-12-02T01:36:00.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:00.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:01.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:01.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:01.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:01.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:01.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:01.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:01.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:01.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:01.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:01.690 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:01.690 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:01.690 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:01.690 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:01.690 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:01.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:01.700 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:01.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:01.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:01.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:01.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:36:01.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:01.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:01.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:01.871 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:01.871 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:01.871 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:01.871 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:01.872 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:01.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:01.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:01.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:01.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:01.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2023-12-02T01:36:01.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:01.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:01.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:01.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:01.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:01.970 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:01.971 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:01.971 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:01.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:01.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:01.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:02.048 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:02.048 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:02.048 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:02.048 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:02.048 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:02.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:02.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:02.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:02.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 1 2023-12-02T01:36:02.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:36:02.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:02.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:02.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:02.229 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:02.229 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:02.229 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:02.229 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:02.229 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:02.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:02.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:02.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:02.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:02.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2023-12-02T01:36:02.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:02.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:02.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:02.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:02.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:02.325 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:02.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:02.404 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:02.404 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:02.404 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:02.404 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:02.404 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:02.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:02.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:02.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:02.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 2 2023-12-02T01:36:02.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:02.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:02.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:02.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:02.580 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:02.580 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:02.580 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:02.580 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:02.580 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:02.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:02.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:02.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:02.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:02.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:02.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2023-12-02T01:36:02.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2023-12-02T01:36:02.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 14 = 0 ']' 2023-12-02T01:36:02.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:02.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 60 + 14 2023-12-02T01:36:02.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=74 2023-12-02T01:36:02.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 74 -gt 0 ']' 2023-12-02T01:36:02.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:02.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:03.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:03.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:03.682 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:03.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:03.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:03.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:03.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:03.774 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:03.775 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:03.775 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:03.775 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:03.775 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:03.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:03.786 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:03.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:03.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:03.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:03.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:03.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:03.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:36:03.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:03.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:03.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:03.952 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:03.952 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:03.952 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:03.952 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:03.952 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:03.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:03.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:04.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:04.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:04.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:04.128 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:04.129 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:04.129 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:04.129 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:04.129 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:04.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.138 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:04.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:04.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:04.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:04.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2023-12-02T01:36:04.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:04.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:04.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:04.304 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:04.305 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:04.305 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:04.305 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:04.305 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:04.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:04.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:04.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:04.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:04.313 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.314 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.314 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.314 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:04.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:04.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:04.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:04.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2023-12-02T01:36:04.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:04.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:04.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:04.479 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:04.479 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:04.480 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:04.480 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:04.480 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:04.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:04.488 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:04.488 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:04.488 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:04.488 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.489 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.489 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.489 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:04.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:04.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:04.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:04.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2023-12-02T01:36:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2023-12-02T01:36:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:04.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:04.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:04.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:04.657 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:04.657 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:04.657 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:04.657 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:04.657 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:04.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:04.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:04.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:04.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 3 2023-12-02T01:36:04.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=19 2023-12-02T01:36:04.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 19 = 0 ']' 2023-12-02T01:36:04.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:04.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 74 + 19 2023-12-02T01:36:04.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=93 2023-12-02T01:36:04.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 93 -gt 0 ']' 2023-12-02T01:36:04.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:04.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:05.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:05.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:05.759 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:05.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:05.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:05.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:05.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:05.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:05.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:05.763 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:05.763 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:05.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:05.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:05.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:05.848 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:05.848 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:05.848 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:05.849 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:05.849 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:05.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:05.857 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:05.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:05.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:05.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:05.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:36:05.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:05.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:05.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:05.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:05.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:05.947 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:05.948 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:05.948 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:05.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:05.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:05.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:06.027 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:06.028 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:06.028 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:06.028 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:06.028 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:06.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.037 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.037 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.037 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:06.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:06.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:06.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:06.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:06.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:06.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:06.203 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:06.203 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:06.203 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 4, 2023-12-02T01:36:06.203 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:06.203 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:06.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:06.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:06.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 4 2023-12-02T01:36:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:06.378 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:06.378 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:06.378 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:06.378 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:06.378 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:06.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:06.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:06.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 3 2023-12-02T01:36:06.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2023-12-02T01:36:06.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:06.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:06.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:06.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:06.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:06.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:06.554 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:06.554 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:06.554 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:06.555 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:06.555 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:06.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:06.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:06.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:06.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2023-12-02T01:36:06.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2023-12-02T01:36:06.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:06.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:06.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:06.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:06.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:06.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:06.731 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:06.731 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:06.731 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:06.731 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:06.731 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:06.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:06.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:06.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:06.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:06.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 2 2023-12-02T01:36:06.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=19 2023-12-02T01:36:06.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 19 = 0 ']' 2023-12-02T01:36:06.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:06.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 93 + 19 2023-12-02T01:36:06.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=112 2023-12-02T01:36:06.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 112 -gt 0 ']' 2023-12-02T01:36:06.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:06.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:07.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:07.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:07.832 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:07.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:07.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:07.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:07.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:07.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:07.836 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:07.836 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:07.836 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:07.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:07.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:07.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:07.921 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:07.921 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:07.921 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:07.921 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:07.921 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:07.930 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:07.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:07.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:08.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:08.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:08.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:36:08.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:08.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:08.099 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:08.099 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:08.099 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:08.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:08.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:08.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:08.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2023-12-02T01:36:08.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:08.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:08.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:08.275 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:08.275 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:08.275 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:08.275 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:08.275 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:08.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:08.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:08.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 3 2023-12-02T01:36:08.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:36:08.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:08.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:08.452 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:08.452 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:08.452 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:08.452 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:08.452 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:08.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:08.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:08.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2023-12-02T01:36:08.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:08.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:08.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:08.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:08.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:08.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:08.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:08.628 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:08.628 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:08.628 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:08.628 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:08.628 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:08.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:08.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:08.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:08.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:08.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 3 2023-12-02T01:36:08.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2023-12-02T01:36:08.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:08.803 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:08.803 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:08.803 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:08.803 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:08.803 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:08.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:08.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:08.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:08.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 3 2023-12-02T01:36:08.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2023-12-02T01:36:08.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2023-12-02T01:36:08.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:08.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 112 + 17 2023-12-02T01:36:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=129 2023-12-02T01:36:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 129 -gt 0 ']' 2023-12-02T01:36:08.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:08.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:09.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:09.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:09.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:09.993 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:09.993 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:09.993 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:09.994 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:09.994 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:10.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:10.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:10.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:10.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:10.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:10.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:10.171 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:10.171 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:10.171 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:10.171 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:10.171 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:10.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:10.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:10.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2023-12-02T01:36:10.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:10.347 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:10.347 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:10.347 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:10.347 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:10.347 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:10.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:10.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:10.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:10.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2023-12-02T01:36:10.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:10.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:10.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:10.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:10.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:10.445 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:10.524 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:10.524 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:10.524 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:10.524 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:10.525 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:10.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:10.532 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:10.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:10.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:10.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2023-12-02T01:36:10.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:10.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:10.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:10.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:10.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:10.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:10.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:10.700 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:10.700 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:10.700 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:10.700 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:10.701 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:10.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:10.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:10.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:10.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:10.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2023-12-02T01:36:10.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:10.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:10.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:10.875 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:10.875 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:10.876 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:10.876 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:10.876 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:10.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:10.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:10.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:10.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:10.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:10.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:10.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:10.885 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:10.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:10.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:10.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:10.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:10.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2023-12-02T01:36:10.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:10.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 10 = 0 ']' 2023-12-02T01:36:10.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:10.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 129 + 10 2023-12-02T01:36:10.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=139 2023-12-02T01:36:10.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 139 -gt 0 ']' 2023-12-02T01:36:10.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:10.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:11.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:11.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:11.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:11.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:11.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:11.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:11.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:11.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:11.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:11.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:11.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:12.065 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:12.065 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:12.065 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:12.065 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:12.065 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:12.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:12.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:12.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:12.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:12.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:36:12.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:12.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:12.242 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:12.243 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:12.243 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:12.243 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:12.243 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:12.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:12.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:12.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:12.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:12.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:12.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:12.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:12.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2023-12-02T01:36:12.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:12.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:12.418 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:12.418 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:12.418 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:12.418 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:12.418 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:12.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:12.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:12.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:12.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:12.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:12.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:12.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:12.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:12.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:12.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:12.593 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:12.593 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:12.593 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:12.593 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:12.593 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:12.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:12.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:12.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2023-12-02T01:36:12.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:36:12.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:12.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:12.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:12.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:12.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:12.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:12.769 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:12.769 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:12.769 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:12.769 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:12.769 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:12.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:12.777 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:12.777 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:12.777 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:12.777 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.777 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.777 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.778 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:12.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:12.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:12.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:12.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2023-12-02T01:36:12.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:12.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:12.944 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:12.944 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:12.944 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:12.945 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:12.945 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:12.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:13.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2023-12-02T01:36:13.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2023-12-02T01:36:13.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 14 = 0 ']' 2023-12-02T01:36:13.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:13.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 139 + 14 2023-12-02T01:36:13.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=153 2023-12-02T01:36:13.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 153 -gt 0 ']' 2023-12-02T01:36:13.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:14.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:14.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:14.044 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:14.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:14.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:14.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:14.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:14.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:14.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:14.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:14.136 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:14.136 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:14.136 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:14.136 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:14.136 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:14.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:14.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:14.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:14.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:14.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:36:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:14.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:14.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:14.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:14.235 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.235 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.235 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:14.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:14.313 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:14.313 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:14.313 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:14.313 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:14.313 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:14.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:14.321 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:14.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:14.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:14.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2023-12-02T01:36:14.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:14.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:14.488 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:14.488 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 3, 2023-12-02T01:36:14.488 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:14.488 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:14.489 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:14.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:14.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:14.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:14.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2023-12-02T01:36:14.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:14.663 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:14.663 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:14.663 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:14.663 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:14.663 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:14.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:14.672 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:14.672 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:14.672 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:14.672 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.672 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.673 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.673 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:14.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:14.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:14.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:14.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2023-12-02T01:36:14.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:14.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:14.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:14.838 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:14.838 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:14.838 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 4, 2023-12-02T01:36:14.838 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:14.838 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:14.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:14.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:14.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:14.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2023-12-02T01:36:14.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2023-12-02T01:36:14.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:14.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:15.013 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:15.013 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:15.013 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:15.013 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:15.013 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:15.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:15.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:15.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 4 2023-12-02T01:36:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=20 2023-12-02T01:36:15.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 20 = 0 ']' 2023-12-02T01:36:15.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:15.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 153 + 20 2023-12-02T01:36:15.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=173 2023-12-02T01:36:15.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 173 -gt 0 ']' 2023-12-02T01:36:15.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:15.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:16.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:16.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:16.115 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:16.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:16.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:16.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:16.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:16.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:16.207 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:16.207 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:16.207 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:16.208 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:16.208 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:16.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:16.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:16.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:36:16.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:16.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:16.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:16.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:16.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:16.308 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.308 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.308 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:16.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:16.386 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:16.386 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:16.386 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:16.386 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:16.386 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:16.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:16.394 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:16.394 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:16.394 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:16.395 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.395 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.395 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:16.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:16.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:16.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:16.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2023-12-02T01:36:16.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:16.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:16.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:16.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:16.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:16.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:16.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:16.563 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:16.563 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 3, 2023-12-02T01:36:16.563 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:16.563 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:16.563 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:16.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:16.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:16.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:16.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2023-12-02T01:36:16.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:16.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:16.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:16.739 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:16.739 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:16.739 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:16.739 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:16.739 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:16.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:16.747 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:16.747 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:16.747 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:16.747 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.747 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.747 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.748 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:16.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:16.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:16.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:16.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2023-12-02T01:36:16.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:16.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:16.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:16.917 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:16.917 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:16.917 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:16.917 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:16.918 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:16.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:16.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:16.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:17.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 3 2023-12-02T01:36:17.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:36:17.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:17.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:17.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:17.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:17.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:17.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:17.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:17.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:17.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:17.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:17.095 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:17.095 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:17.095 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:17.095 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:17.096 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:17.104 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:17.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:17.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:17.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:17.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:17.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 2 2023-12-02T01:36:17.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2023-12-02T01:36:17.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2023-12-02T01:36:17.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:17.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 173 + 17 2023-12-02T01:36:17.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=190 2023-12-02T01:36:17.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 190 -gt 0 ']' 2023-12-02T01:36:17.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:17.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:18.196 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:18.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:18.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:18.286 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:18.286 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:18.287 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:18.287 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:18.287 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.296 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:18.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:18.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:18.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:36:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:18.466 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:18.466 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:18.466 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:18.467 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:18.467 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:18.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:18.475 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:18.475 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:18.475 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:18.475 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.475 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.475 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:18.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:18.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:18.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:18.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2023-12-02T01:36:18.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:18.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:18.641 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:18.641 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:18.642 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:18.642 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:18.642 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:18.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:18.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:18.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2023-12-02T01:36:18.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:36:18.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:18.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:18.815 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:18.815 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:18.815 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:18.815 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:18.815 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:18.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:18.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:18.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:18.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:18.824 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.824 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.824 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.824 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:18.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:18.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:18.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:18.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 0 2023-12-02T01:36:18.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:36:18.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:18.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:18.992 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:18.992 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:18.992 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:18.992 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:18.992 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:19.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:19.000 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:19.000 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:19.000 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:19.000 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:19.000 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:19.001 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:19.001 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:19.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:19.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:19.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:19.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:19.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2023-12-02T01:36:19.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:19.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:19.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:19.166 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:19.166 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:19.167 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:19.167 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:19.167 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:19.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:19.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:19.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:19.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:19.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:19.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2023-12-02T01:36:19.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:19.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2023-12-02T01:36:19.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:19.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 190 + 12 2023-12-02T01:36:19.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=202 2023-12-02T01:36:19.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 202 -gt 0 ']' 2023-12-02T01:36:19.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:19.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:20.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:20.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:20.267 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:20.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:20.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:20.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:20.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:20.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:20.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:20.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:20.355 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:20.355 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:20.355 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 4, 2023-12-02T01:36:20.355 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:20.355 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.364 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:20.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:20.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:36:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:20.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:20.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:20.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:20.454 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.454 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:20.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:20.531 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:20.531 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:20.531 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:20.531 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:20.531 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:20.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:20.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:20.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:20.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:20.539 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.540 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.540 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.540 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:20.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:20.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:20.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:20.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:20.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:20.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:20.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:20.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:20.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:20.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:20.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:20.708 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:20.709 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:20.709 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:20.709 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:20.709 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:20.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:20.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 0 2023-12-02T01:36:20.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:20.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:20.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:20.883 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:20.883 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:20.883 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:20.883 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:20.883 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:20.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:20.891 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:20.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:20.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:20.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2023-12-02T01:36:20.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:20.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:20.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:21.059 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:21.059 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:21.059 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:21.059 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:21.059 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:21.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:21.068 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:21.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:21.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:21.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:21.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2023-12-02T01:36:21.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:21.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:21.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:21.236 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:21.236 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:21.236 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:21.236 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:21.236 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:21.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:21.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:21.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:21.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:21.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:21.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 3 2023-12-02T01:36:21.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:36:21.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2023-12-02T01:36:21.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:21.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 202 + 15 2023-12-02T01:36:21.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=217 2023-12-02T01:36:21.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 217 -gt 0 ']' 2023-12-02T01:36:21.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:21.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:22.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:22.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:22.336 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:22.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:22.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:22.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:22.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:22.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:22.340 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.340 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.340 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:22.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:22.427 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:22.428 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:22.428 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 4, 2023-12-02T01:36:22.428 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:22.428 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:22.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:22.437 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:22.437 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:22.437 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:22.437 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.437 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.437 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.438 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:22.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:22.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:22.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:22.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:36:22.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:22.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:22.605 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:22.605 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:22.605 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:22.605 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:22.605 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:22.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:22.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:22.780 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:22.781 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:22.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:22.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:22.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:22.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 0 2023-12-02T01:36:22.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:22.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:22.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:22.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:22.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:22.955 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:22.955 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:22.955 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:22.955 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:22.955 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:22.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:22.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:23.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:23.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:23.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2023-12-02T01:36:23.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:23.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:23.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:23.132 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:23.132 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:23.133 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:23.133 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:23.133 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:23.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:23.142 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:23.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:23.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:23.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:23.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:23.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2023-12-02T01:36:23.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2023-12-02T01:36:23.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:23.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:23.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:23.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:23.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:23.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:23.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:23.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:23.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:23.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:23.308 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:23.308 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:23.308 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:23.308 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:23.308 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:23.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:23.316 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:23.316 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:23.317 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:23.317 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:23.317 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:23.317 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:23.317 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:23.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:23.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:23.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:23.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:23.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2023-12-02T01:36:23.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2023-12-02T01:36:23.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 16 = 0 ']' 2023-12-02T01:36:23.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:23.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 217 + 16 2023-12-02T01:36:23.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=233 2023-12-02T01:36:23.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 233 -gt 0 ']' 2023-12-02T01:36:23.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:23.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:24.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:24.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:24.409 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:24.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:24.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:24.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:24.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:24.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:24.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:24.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:24.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:24.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:24.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:24.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:24.500 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:24.500 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:24.500 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:24.500 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:24.500 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:24.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:24.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:24.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:24.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:24.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:24.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:24.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:24.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:24.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:36:24.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:24.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:24.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:24.676 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:24.677 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:24.677 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:24.677 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:24.677 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:24.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:24.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:24.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:24.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2023-12-02T01:36:24.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:24.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:24.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:24.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:24.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:24.773 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:24.773 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:24.773 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:24.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:24.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:24.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:24.853 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:24.853 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:24.853 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:24.853 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:24.853 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:24.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:24.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:24.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:24.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:24.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:24.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:24.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2023-12-02T01:36:24.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:24.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:24.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:24.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:24.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:24.948 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:24.949 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:24.949 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:24.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:24.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:24.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:25.026 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:25.026 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:25.027 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:25.027 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:25.027 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:25.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:25.036 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:25.036 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:25.036 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:25.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:25.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:25.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:25.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:25.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2023-12-02T01:36:25.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:25.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:25.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:25.203 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:25.203 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:25.203 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:25.203 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:25.203 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:25.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:25.210 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:25.211 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:25.211 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:25.211 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:25.211 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:25.212 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:25.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:25.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:25.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:25.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:25.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:25.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2023-12-02T01:36:25.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:36:25.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:25.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:25.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:25.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:25.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:25.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:25.301 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:25.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:25.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:25.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:25.378 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:25.378 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:25.378 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:25.378 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:25.379 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:25.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:25.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:25.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:25.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 4 2023-12-02T01:36:25.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=19 2023-12-02T01:36:25.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 19 = 0 ']' 2023-12-02T01:36:25.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:25.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 233 + 19 2023-12-02T01:36:25.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=252 2023-12-02T01:36:25.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 252 -gt 0 ']' 2023-12-02T01:36:25.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:25.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:26.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:26.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:26.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:26.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:26.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:26.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:26.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:26.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:26.481 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:26.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:26.566 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:26.566 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:26.566 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:26.566 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:26.566 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:26.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:26.576 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:26.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:26.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:26.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:26.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:26.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:26.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:26.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:26.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:26.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:26.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:26.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:26.742 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:26.742 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:26.742 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:26.743 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:26.743 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:26.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:26.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:26.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:26.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:36:26.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:26.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:26.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:26.916 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:26.917 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:26.917 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:26.917 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:26.917 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:26.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:26.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:27.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:27.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:27.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2023-12-02T01:36:27.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:27.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:27.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:27.092 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:27.092 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:27.092 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:27.092 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:27.092 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:27.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:27.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:27.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:27.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 1 2023-12-02T01:36:27.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:27.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:27.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:27.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:27.269 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:27.269 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:27.270 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:27.270 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:27.270 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:27.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:27.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:27.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:27.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:27.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2023-12-02T01:36:27.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:27.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:27.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:27.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:27.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:27.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:27.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:27.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:27.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:27.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:27.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:27.443 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:27.443 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:27.443 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:27.444 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:27.444 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:27.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:27.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:27.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:27.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:27.452 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:27.452 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:27.453 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:27.453 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:27.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:27.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:27.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:27.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:27.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2023-12-02T01:36:27.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:27.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 11 = 0 ']' 2023-12-02T01:36:27.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:27.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 252 + 11 2023-12-02T01:36:27.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=263 2023-12-02T01:36:27.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 263 -gt 0 ']' 2023-12-02T01:36:27.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:27.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:28.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:28.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:28.544 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:28.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:28.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:28.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:28.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:28.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:28.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:28.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:28.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:28.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:28.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:28.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:28.630 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:28.630 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:28.630 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:28.630 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:28.630 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:28.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:28.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:28.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:28.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:28.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:28.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:28.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:28.806 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:28.806 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:28.806 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:28.806 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:28.807 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:28.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:28.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:28.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:28.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:28.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2023-12-02T01:36:28.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:28.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:28.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:28.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:28.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:28.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:28.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:28.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:28.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:28.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:28.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:28.981 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:28.981 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:28.981 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:28.981 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:28.981 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:28.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:29.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:29.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:29.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:29.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:29.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:29.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:29.158 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:29.158 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:29.158 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:29.159 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:29.159 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:29.167 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:29.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:29.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:29.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:29.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:29.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2023-12-02T01:36:29.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:29.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:29.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:29.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:29.333 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:29.333 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:29.334 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:29.334 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:29.334 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:29.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:29.343 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:29.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2023-12-02T01:36:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:29.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:29.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:29.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:29.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:29.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:29.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:29.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:29.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:29.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:29.510 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:29.510 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:29.510 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:29.510 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:29.510 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:29.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:29.518 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:29.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:29.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:29.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:29.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2023-12-02T01:36:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 10 = 0 ']' 2023-12-02T01:36:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:29.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 263 + 10 2023-12-02T01:36:29.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=273 2023-12-02T01:36:29.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 273 -gt 0 ']' 2023-12-02T01:36:29.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:29.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:30.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:30.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:30.613 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:30.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:30.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:30.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:30.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:30.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:30.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:30.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:30.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:30.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:30.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:30.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:30.703 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:30.703 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:30.703 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:30.703 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:30.703 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:30.714 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:30.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:30.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:30.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:30.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:30.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:30.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:30.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:30.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:30.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:30.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:30.880 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:30.881 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:30.881 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:30.881 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:30.881 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:30.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:30.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:30.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:30.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:30.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 3 2023-12-02T01:36:30.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:30.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:30.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:30.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:30.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:30.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:31.056 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:31.056 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:31.056 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:31.056 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:31.056 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:31.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:31.065 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:31.065 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:31.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:31.066 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:31.066 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:31.066 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:31.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:31.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:31.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:31.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:31.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:31.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 0 2023-12-02T01:36:31.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:31.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:31.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:31.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:31.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:31.153 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:31.153 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:31.153 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:31.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:31.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:31.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:31.232 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:31.232 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:31.232 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:31.232 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:31.232 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:31.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:31.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:31.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:31.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 1 2023-12-02T01:36:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:31.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:31.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:31.406 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:31.406 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:31.406 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:31.406 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:31.406 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:31.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:31.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:31.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:31.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 1 2023-12-02T01:36:31.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:31.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:31.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:31.583 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:31.583 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:31.583 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:31.583 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:31.583 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:31.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:31.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:31.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:31.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:31.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2023-12-02T01:36:31.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:31.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 7 = 0 ']' 2023-12-02T01:36:31.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:31.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 273 + 7 2023-12-02T01:36:31.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=280 2023-12-02T01:36:31.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 280 -gt 0 ']' 2023-12-02T01:36:31.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:31.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:32.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:32.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:32.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:32.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:32.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:32.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:32.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:32.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:32.775 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:32.775 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:32.775 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:32.775 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:32.775 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:32.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:32.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:32.953 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:32.953 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:32.954 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:32.954 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:32.954 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:32.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:32.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:32.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:33.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 2 2023-12-02T01:36:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:33.127 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:33.128 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:33.128 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:33.128 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:33.128 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:33.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:33.136 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:33.136 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:33.136 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:33.136 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.136 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:33.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:33.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:33.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:33.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 1 2023-12-02T01:36:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:33.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:33.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:33.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:33.305 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:33.305 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:33.306 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:33.306 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:33.306 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:33.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:33.314 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:33.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:33.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:33.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:33.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:33.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:33.481 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:33.481 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:33.481 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:33.481 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:33.481 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:33.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:33.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:33.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2023-12-02T01:36:33.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:33.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:33.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:33.658 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:33.658 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:33.658 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:33.658 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:33.658 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:33.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:33.668 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:33.668 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:33.668 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:33.668 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:33.669 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:33.669 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:33.669 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:33.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:33.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:33.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:33.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:33.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 0 2023-12-02T01:36:33.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:33.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 9 = 0 ']' 2023-12-02T01:36:33.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:33.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 280 + 9 2023-12-02T01:36:33.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=289 2023-12-02T01:36:33.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 289 -gt 0 ']' 2023-12-02T01:36:33.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:33.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:34.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:34.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:34.760 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:34.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:34.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:34.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:34.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:34.764 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:34.764 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:34.764 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:34.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:34.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:34.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:34.852 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:34.852 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:34.852 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:34.852 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:34.852 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:34.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:34.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:34.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:34.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:34.862 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:34.863 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:34.863 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:34.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:34.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:34.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:34.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:34.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:34.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:36:34.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:34.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:34.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:34.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:35.030 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:35.030 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:35.030 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:35.031 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:35.031 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:35.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:35.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:35.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2023-12-02T01:36:35.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:35.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:35.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:35.206 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:35.209 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:35.209 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:35.209 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:35.209 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:35.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:35.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:35.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:35.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 3 2023-12-02T01:36:35.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:36:35.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:35.382 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:35.382 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 3, 2023-12-02T01:36:35.382 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:35.382 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:35.382 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:35.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:35.390 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:35.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:35.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:35.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2023-12-02T01:36:35.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:35.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:35.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:35.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:35.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:35.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:35.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:35.557 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:35.557 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:35.557 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:35.557 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:35.557 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:35.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:35.565 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:35.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:35.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:35.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2023-12-02T01:36:35.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:36:35.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:35.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:35.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:35.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:35.654 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.654 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.655 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:35.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:35.733 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:35.733 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:35.733 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:35.733 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:35.733 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:35.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:35.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:35.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 0 2023-12-02T01:36:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:36:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2023-12-02T01:36:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:35.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 289 + 15 2023-12-02T01:36:35.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=304 2023-12-02T01:36:35.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 304 -gt 0 ']' 2023-12-02T01:36:35.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:35.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:36.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:36.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:36.834 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:36.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:36.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:36.919 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:36.919 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:36.919 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:36.919 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:36.919 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:36.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:36.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:36.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:37.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:37.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:37.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:36:37.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:37.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:37.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:37.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:37.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:37.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:37.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:37.096 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:37.096 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:37.096 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:37.096 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:37.096 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:37.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:37.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:37.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2023-12-02T01:36:37.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:37.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:37.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:37.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:37.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:37.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:37.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:37.273 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:37.273 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:37.274 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 4, 2023-12-02T01:36:37.274 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:37.274 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:37.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:37.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:37.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 4 2023-12-02T01:36:37.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:37.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:37.447 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:37.448 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:37.448 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:37.448 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:37.448 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:37.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:37.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:37.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:37.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:37.456 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.457 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.457 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.457 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:37.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:37.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:37.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:37.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2023-12-02T01:36:37.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:37.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:37.625 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:37.625 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:37.625 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:37.625 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:37.625 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:37.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:37.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:37.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:37.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2023-12-02T01:36:37.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:36:37.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:37.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:37.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:37.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:37.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:37.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:37.801 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:37.801 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:37.802 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:37.802 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:37.802 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:37.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:37.811 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:37.811 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:37.811 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:37.812 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:37.812 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:37.812 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:37.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:37.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:37.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:37.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:37.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:37.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 0 2023-12-02T01:36:37.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2023-12-02T01:36:37.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2023-12-02T01:36:37.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:37.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 304 + 15 2023-12-02T01:36:37.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=319 2023-12-02T01:36:37.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 319 -gt 0 ']' 2023-12-02T01:36:37.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:37.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:38.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:38.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:38.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:38.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:38.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:38.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:38.994 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:38.994 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:38.994 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:38.994 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:38.994 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:39.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:39.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:39.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:39.004 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:39.004 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.004 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.004 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.004 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:39.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:39.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:39.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:39.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:36:39.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:39.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:39.173 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:39.173 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:39.173 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:39.173 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:39.173 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:39.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:39.181 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:39.181 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:39.181 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:39.182 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.182 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.182 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.182 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:39.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:39.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:39.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:39.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2023-12-02T01:36:39.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:39.348 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:39.348 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:39.348 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:39.348 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:39.348 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:39.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:39.356 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:39.356 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.357 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:39.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:39.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:39.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:39.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 3 2023-12-02T01:36:39.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2023-12-02T01:36:39.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:39.521 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:39.522 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:39.522 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:39.522 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:39.522 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:39.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:39.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:39.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:39.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2023-12-02T01:36:39.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:39.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:39.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:39.700 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:39.700 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:39.700 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:39.700 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:39.700 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:39.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:39.708 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:39.708 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:39.708 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:39.708 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.708 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.709 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.709 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:39.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:39.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:39.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:39.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 3 2023-12-02T01:36:39.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2023-12-02T01:36:39.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:39.874 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:39.874 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:39.874 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:39.874 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:39.874 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:39.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:39.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:39.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:39.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:39.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 1 2023-12-02T01:36:39.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2023-12-02T01:36:39.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 14 = 0 ']' 2023-12-02T01:36:39.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:39.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 319 + 14 2023-12-02T01:36:39.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=333 2023-12-02T01:36:39.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 333 -gt 0 ']' 2023-12-02T01:36:39.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:39.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:40.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:40.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:40.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:40.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:40.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:40.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:40.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:40.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:40.979 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:40.979 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:40.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:40.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:40.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:40.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:41.062 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:41.062 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:41.062 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:41.062 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:41.062 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:41.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:41.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:41.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:36:41.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:41.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:41.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:41.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:41.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:41.159 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.160 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.160 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:41.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:41.239 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:41.239 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:41.239 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:41.239 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:41.239 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:41.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:41.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:41.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:41.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2023-12-02T01:36:41.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:41.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:41.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:41.413 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:41.413 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:41.413 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:41.413 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:41.413 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:41.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:41.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:41.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:41.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 1 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.513 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:41.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:41.591 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:41.591 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:41.591 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:41.591 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:41.591 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:41.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:41.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:41.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:41.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2023-12-02T01:36:41.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:41.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:41.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:41.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:41.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:41.688 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:41.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:41.767 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:41.767 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:41.767 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:41.767 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:41.767 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:41.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:41.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:41.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:41.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:41.775 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.775 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.776 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:41.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:41.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:41.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:41.942 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:41.943 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:41.943 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:41.943 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:41.943 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:41.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:41.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:42.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:42.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:42.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 1 2023-12-02T01:36:42.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2023-12-02T01:36:42.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 11 = 0 ']' 2023-12-02T01:36:42.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:42.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 333 + 11 2023-12-02T01:36:42.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=344 2023-12-02T01:36:42.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 344 -gt 0 ']' 2023-12-02T01:36:42.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:42.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:43.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:43.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:43.044 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:43.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:43.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:43.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:43.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:43.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:43.133 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:43.134 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:43.134 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:43.134 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:43.134 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:43.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:43.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:43.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:43.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:36:43.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:43.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:43.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:43.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:43.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:43.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:43.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:43.310 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:43.310 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:43.310 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:43.310 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:43.310 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:43.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:43.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:43.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 0 2023-12-02T01:36:43.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:43.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:43.485 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:43.486 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:43.486 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:43.486 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:43.486 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:43.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:43.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:43.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:43.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:43.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 0 2023-12-02T01:36:43.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:43.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:43.662 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:43.662 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 3, 2023-12-02T01:36:43.662 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:43.662 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:43.662 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:43.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2023-12-02T01:36:43.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2023-12-02T01:36:43.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2023-12-02T01:36:43.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:43.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:43.840 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:43.840 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:43.840 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:43.841 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:43.841 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:43.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:43.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:43.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:43.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2023-12-02T01:36:43.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2023-12-02T01:36:43.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:43.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:43.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:43.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:43.937 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:43.938 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:43.938 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:43.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:43.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:43.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:44.016 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:44.016 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:44.016 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:44.016 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:44.016 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:44.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:44.025 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:44.025 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:44.025 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:44.025 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:44.025 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:44.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:44.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:44.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:44.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:44.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2023-12-02T01:36:44.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2023-12-02T01:36:44.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 10 = 0 ']' 2023-12-02T01:36:44.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:44.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 344 + 10 2023-12-02T01:36:44.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=354 2023-12-02T01:36:44.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 354 -gt 0 ']' 2023-12-02T01:36:44.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:44.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:45.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:45.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:45.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:45.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:45.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:45.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:45.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:45.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:45.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:45.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:45.207 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:45.208 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:45.208 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 3, 2023-12-02T01:36:45.208 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:45.208 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:45.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2023-12-02T01:36:45.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2023-12-02T01:36:45.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2023-12-02T01:36:45.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:45.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:45.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:45.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:45.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:45.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:45.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:45.384 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:45.385 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:45.385 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:45.385 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:45.385 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:45.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:45.394 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:45.394 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:45.394 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:45.395 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.395 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.395 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:45.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:45.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:45.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:45.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 1 2023-12-02T01:36:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:45.560 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:45.560 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:45.560 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:45.560 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:45.561 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:45.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.570 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.570 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.570 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:45.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:45.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:45.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:45.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 0 2023-12-02T01:36:45.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:45.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:45.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:45.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:45.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:45.659 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.659 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:45.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:45.738 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:45.738 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:45.738 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:45.738 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:45.738 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:45.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:45.746 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:45.746 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:45.746 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:45.747 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.747 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.747 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.747 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:45.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:45.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:45.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:45.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2023-12-02T01:36:45.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:45.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:45.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:45.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:45.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:45.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.836 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:45.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:45.913 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:45.914 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:45.914 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:45.914 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:45.914 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:45.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:45.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:46.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:46.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:46.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2023-12-02T01:36:46.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:46.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:46.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:46.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:46.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:46.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:46.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:46.010 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:46.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:46.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:46.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:46.087 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:46.087 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:46.087 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:46.088 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:46.088 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:46.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:46.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:46.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:46.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:46.097 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:46.098 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:46.098 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:46.098 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:46.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:46.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:46.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:46.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:46.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 0 2023-12-02T01:36:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2023-12-02T01:36:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 7 = 0 ']' 2023-12-02T01:36:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:46.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 354 + 7 2023-12-02T01:36:46.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=361 2023-12-02T01:36:46.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 361 -gt 0 ']' 2023-12-02T01:36:46.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:46.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:47.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:47.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:47.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:47.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:47.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:47.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:47.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:47.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:47.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:47.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:47.280 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:47.280 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:47.280 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:47.280 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:47.280 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:47.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:47.289 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:47.289 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:47.289 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:47.289 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.290 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.290 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.290 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:47.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:47.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:47.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:47.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2023-12-02T01:36:47.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:47.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:47.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:47.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:47.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:47.459 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:47.459 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:47.459 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:47.459 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:47.459 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:47.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:47.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:47.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:47.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:47.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2023-12-02T01:36:47.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:47.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:47.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:47.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:47.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:47.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.557 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.557 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:47.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:47.634 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:47.634 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:47.635 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:47.635 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:47.635 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:47.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:47.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:47.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2023-12-02T01:36:47.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:47.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:47.809 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:47.809 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:47.809 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:47.809 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:47.809 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:47.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:47.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:47.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:47.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:47.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:47.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:47.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:47.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 1 2023-12-02T01:36:47.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:47.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:47.984 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:47.984 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:47.984 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:47.984 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:47.984 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:47.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:47.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:48.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:48.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:48.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 0 2023-12-02T01:36:48.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:48.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:48.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:48.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:48.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:48.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:48.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:48.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:48.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:48.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:48.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:48.162 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:48.162 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:48.162 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:48.162 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:48.162 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:48.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:48.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:48.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:48.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:48.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 0 2023-12-02T01:36:48.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2023-12-02T01:36:48.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 4 = 0 ']' 2023-12-02T01:36:48.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:48.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 361 + 4 2023-12-02T01:36:48.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=365 2023-12-02T01:36:48.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 365 -gt 0 ']' 2023-12-02T01:36:48.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:48.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:49.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:49.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:49.263 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:49.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:49.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:49.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:49.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:49.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:49.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:49.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:49.354 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:49.354 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:49.354 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:49.354 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:49.354 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:49.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:49.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:49.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:49.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:49.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:49.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:49.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:49.529 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:49.529 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:49.529 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:49.529 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:49.529 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:49.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:49.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:49.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:49.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2023-12-02T01:36:49.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:49.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:49.705 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:49.705 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:49.705 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:49.706 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:49.706 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:49.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:49.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:49.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:49.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2023-12-02T01:36:49.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:49.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:49.879 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:49.879 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:49.879 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:49.879 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:49.879 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:49.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:49.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:49.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2023-12-02T01:36:49.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:49.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:49.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:49.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:49.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:49.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:49.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:49.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:49.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:49.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:49.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:50.054 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:50.054 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:50.055 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:50.055 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:50.055 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:50.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:50.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:50.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:50.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:50.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2023-12-02T01:36:50.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:50.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:50.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:50.232 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:50.232 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:50.232 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:50.233 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:50.233 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:50.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:50.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:50.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:50.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:50.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:50.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:50.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:50.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:50.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:50.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:50.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:50.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:50.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2023-12-02T01:36:50.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:50.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 1 = 0 ']' 2023-12-02T01:36:50.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2023-12-02T01:36:50.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 365 + 1 2023-12-02T01:36:50.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=366 2023-12-02T01:36:50.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 366 -gt 0 ']' 2023-12-02T01:36:50.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:51.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:51.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:51.333 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:51.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:51.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:51.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:51.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:51.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:51.336 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.336 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.337 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:51.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:51.423 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:51.423 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:51.423 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:51.423 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:51.423 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:51.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:51.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:51.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:51.435 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:51.435 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.435 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.435 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.435 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:51.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:51.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:51.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:51.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:51.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:51.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:51.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:51.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:51.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:51.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:51.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:51.599 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:51.599 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:51.599 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:51.599 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:51.599 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:51.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.609 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.609 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:51.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:51.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:51.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:51.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:51.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:51.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:51.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:51.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:51.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:51.695 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.695 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.695 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:51.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:51.774 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:51.774 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:51.774 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:51.774 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:51.774 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:51.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:51.782 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:51.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:51.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:51.950 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:51.950 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:51.951 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:51.951 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:51.951 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:51.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:51.958 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:51.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:52.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:52.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:52.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:52.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:52.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:52.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:52.129 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:52.129 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:52.129 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:52.130 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:52.130 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:52.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:52.138 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:52.138 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:52.138 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:52.138 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:52.139 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:52.139 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:52.139 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:52.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:52.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:52.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:52.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:52.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:52.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:52.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:52.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:52.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:52.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:52.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:52.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:52.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:52.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:52.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:52.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:52.304 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:52.304 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:52.304 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:52.304 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:52.304 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:52.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:52.312 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:52.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:52.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:52.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:52.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:52.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:52.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2023-12-02T01:36:52.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:52.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=1 2023-12-02T01:36:52.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 1 -gt 3 ']' 2023-12-02T01:36:52.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 366 + 0 2023-12-02T01:36:52.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=366 2023-12-02T01:36:52.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 366 -gt 0 ']' 2023-12-02T01:36:52.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:52.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:53.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:53.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:53.404 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:53.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:53.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:53.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:53.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:53.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:53.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:53.492 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:53.492 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:53.492 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:53.492 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:53.492 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:53.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:53.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:53.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:53.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:53.502 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:53.502 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:53.502 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:53.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:53.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:53.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:53.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:53.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:53.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:53.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:53.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:53.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:53.670 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:53.670 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:53.670 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:53.670 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:53.670 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:53.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:53.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:53.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:53.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:53.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:53.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:53.846 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:53.846 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:53.846 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:53.846 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:53.846 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:53.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:53.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:53.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:53.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:53.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:53.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:53.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:53.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:53.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:53.943 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:53.943 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:53.943 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:53.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:53.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:53.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:54.021 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:54.021 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:54.021 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:54.021 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:54.022 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:54.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:54.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:54.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:54.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:54.030 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:54.030 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:54.031 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:54.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:54.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:54.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:54.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:54.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:54.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:54.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:54.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:54.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:54.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:54.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:54.119 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:54.119 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:54.119 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:54.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:54.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:54.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:54.197 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:54.198 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:54.198 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:54.198 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:54.198 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:54.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:54.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:54.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:54.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:54.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:54.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:54.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:54.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:54.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:54.373 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:54.373 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:54.373 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:54.373 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:54.373 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:54.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:54.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:54.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:54.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2023-12-02T01:36:54.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 1 + 1 2023-12-02T01:36:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=2 2023-12-02T01:36:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2023-12-02T01:36:54.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 366 + 0 2023-12-02T01:36:54.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=366 2023-12-02T01:36:54.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 366 -gt 0 ']' 2023-12-02T01:36:54.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:55.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:55.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:55.474 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:55.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:55.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:55.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:55.564 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:55.564 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:55.564 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:55.564 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:55.564 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:55.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:55.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:55.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:55.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:55.573 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:55.574 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:55.574 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:55.574 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:55.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:55.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:55.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:55.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:55.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:55.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:55.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:55.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:55.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:55.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:55.664 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:55.664 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:55.664 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:55.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:55.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:55.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:55.743 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:55.743 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:55.743 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:55.743 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:55.743 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:55.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:55.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:55.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:55.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:55.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:55.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:55.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:55.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:55.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:55.918 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:55.919 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:55.919 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:55.919 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:55.919 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:55.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:55.927 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:55.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:56.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:56.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:56.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:56.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:56.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:56.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:56.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:56.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:56.018 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:56.018 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:56.018 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:56.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:56.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:56.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:56.096 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:56.096 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:56.096 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:56.096 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:56.096 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:56.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:56.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:56.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:56.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2023-12-02T01:36:56.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:56.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:56.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:56.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:56.271 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:56.271 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:36:56.271 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:56.271 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:56.271 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:56.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:56.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:56.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:56.280 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:56.280 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:56.280 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:56.280 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:56.280 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:56.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:56.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:56.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:56.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:56.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2023-12-02T01:36:56.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:36:56.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:56.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:56.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:56.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:56.367 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:56.368 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:56.368 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:56.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:56.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:56.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:56.446 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:56.446 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:56.446 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:56.446 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:56.446 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:56.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:56.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:56.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:56.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:56.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2023-12-02T01:36:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 3 = 0 ']' 2023-12-02T01:36:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2023-12-02T01:36:56.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 366 + 3 2023-12-02T01:36:56.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=369 2023-12-02T01:36:56.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 369 -gt 0 ']' 2023-12-02T01:36:56.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:56.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:57.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:57.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:57.546 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:57.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:57.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:57.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:57.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:57.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:57.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:57.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:57.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:57.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:57.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:57.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:57.636 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:57.636 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:57.636 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:57.636 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:57.636 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:57.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:57.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:57.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:57.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:57.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:57.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:57.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:57.816 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:57.817 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:57.817 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:57.817 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:57.817 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:57.826 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:57.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:57.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:57.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:57.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:57.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:57.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:57.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:57.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:57.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:57.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:57.916 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:57.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:57.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:57.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:57.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:57.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:57.994 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:57.994 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:57.994 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:36:57.994 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:57.994 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:58.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:58.002 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:58.002 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:58.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:58.003 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:58.003 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:58.003 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:58.003 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:58.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:58.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:36:58.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:58.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 2 2023-12-02T01:36:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:58.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:58.168 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:58.168 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:58.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:36:58.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:58.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:58.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2023-12-02T01:36:58.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:36:58.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:58.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:58.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:58.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:58.264 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:58.264 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:58.264 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:58.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:58.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:58.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:58.341 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:58.342 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:36:58.342 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:58.342 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:58.342 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:36:58.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:58.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:36:58.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:36:58.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:36:58.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2023-12-02T01:36:58.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:36:58.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:58.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:58.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:58.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:58.437 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:58.437 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:58.437 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:58.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:58.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:58.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:58.514 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:58.514 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:58.514 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:58.515 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:58.515 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:36:58.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:58.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:36:58.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:58.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:58.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 1 2023-12-02T01:36:58.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:36:58.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 6 = 0 ']' 2023-12-02T01:36:58.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2023-12-02T01:36:58.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 369 + 6 2023-12-02T01:36:58.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=375 2023-12-02T01:36:58.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 375 -gt 0 ']' 2023-12-02T01:36:58.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:36:58.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:36:59.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:36:59.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:36:59.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:36:59.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:36:59.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:59.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:59.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:59.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:59.618 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:59.619 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:59.619 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:59.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:59.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:59.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:59.705 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:59.705 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:59.705 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:36:59.705 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:59.705 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:59.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:59.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:36:59.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:36:59.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:36:59.714 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:59.715 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:59.715 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:59.715 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:36:59.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:59.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:36:59.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:36:59.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:36:59.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:36:59.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:36:59.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:59.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:59.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:59.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:59.803 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:59.803 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:59.803 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:59.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:59.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:59.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:59.883 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:36:59.883 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:36:59.883 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:36:59.883 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:36:59.883 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:36:59.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:36:59.892 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:36:59.892 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:36:59.893 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:36:59.893 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:59.893 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:59.893 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:59.893 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:36:59.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:59.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:36:59.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:36:59.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:36:59.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:36:59.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:36:59.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:36:59.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:36:59.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:36:59.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:36:59.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:36:59.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:36:59.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:36:59.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:36:59.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:36:59.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:00.059 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:00.059 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:00.059 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 2, 2023-12-02T01:37:00.059 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:00.059 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:00.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:00.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:00.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:37:00.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:37:00.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 2 2023-12-02T01:37:00.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:37:00.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:00.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:00.233 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:00.233 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:00.233 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:00.233 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:00.233 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:00.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:00.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:00.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:00.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:00.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2023-12-02T01:37:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:37:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:00.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:00.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:00.407 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:00.408 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 2, 2023-12-02T01:37:00.408 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:00.408 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:00.408 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:00.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:00.417 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:00.417 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:00.417 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:00.418 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:00.418 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:00.418 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:00.418 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:00.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:00.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:00.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2023-12-02T01:37:00.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2023-12-02T01:37:00.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2023-12-02T01:37:00.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2023-12-02T01:37:00.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:00.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:00.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:00.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:00.504 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:00.504 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:00.504 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:00.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:00.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:00.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:00.583 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:00.583 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:00.583 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:37:00.583 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:00.583 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:00.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:00.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:00.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:37:00.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:37:00.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 1 2023-12-02T01:37:00.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2023-12-02T01:37:00.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 6 = 0 ']' 2023-12-02T01:37:00.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2023-12-02T01:37:00.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 375 + 6 2023-12-02T01:37:00.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=381 2023-12-02T01:37:00.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 381 -gt 0 ']' 2023-12-02T01:37:00.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:37:00.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:37:01.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:37:01.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:37:01.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:37:01.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:37:01.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:01.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:37:01.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:37:01.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:37:01.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:01.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:01.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:01.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:37:01.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:01.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:37:01.770 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:01.770 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:01.770 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:01.770 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:01.770 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:01.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:37:01.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:01.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:37:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:01.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:01.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:01.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:37:01.945 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:01.945 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:01.945 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:37:01.945 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:01.945 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:01.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:01.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:37:01.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:37:01.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:37:01.954 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:01.954 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:01.954 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:01.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:37:01.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:01.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:37:02.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:37:02.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:37:02.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2023-12-02T01:37:02.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2023-12-02T01:37:02.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:02.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:37:02.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:37:02.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:37:02.043 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.043 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.043 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:37:02.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:02.122 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:02.122 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:02.122 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 1, 2023-12-02T01:37:02.122 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:02.122 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:37:02.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:02.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:37:02.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:37:02.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2023-12-02T01:37:02.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:37:02.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:02.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:02.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:02.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:02.220 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.220 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.220 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:02.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:02.297 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:02.297 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:02.298 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:02.298 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:02.298 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:02.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:02.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:02.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 0 2023-12-02T01:37:02.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2023-12-02T01:37:02.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:02.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:02.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:02.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:02.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:02.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:02.473 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:02.473 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 1, 2023-12-02T01:37:02.473 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:02.473 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:02.473 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:02.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:02.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:02.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:02.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:02.483 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.483 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.483 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:02.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:02.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2023-12-02T01:37:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2023-12-02T01:37:02.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2023-12-02T01:37:02.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:37:02.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:02.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:02.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:02.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:02.570 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.570 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.570 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:02.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:02.645 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:02.646 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:02.646 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:02.646 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:02.646 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:02.655 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:02.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:02.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:02.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:02.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:02.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2023-12-02T01:37:02.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2023-12-02T01:37:02.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 3 = 0 ']' 2023-12-02T01:37:02.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2023-12-02T01:37:02.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 381 + 3 2023-12-02T01:37:02.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=384 2023-12-02T01:37:02.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 384 -gt 0 ']' 2023-12-02T01:37:02.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:37:02.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:37:03.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:37:03.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:37:03.744 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:37:03.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:37:03.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:03.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:37:03.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:37:03.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:37:03.748 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:03.748 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:03.748 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:03.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:37:03.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:03.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:37:03.835 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:03.835 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:03.835 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:03.835 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:03.835 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:37:03.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:03.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:37:03.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:03.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:03.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:03.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:03.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:03.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:37:04.013 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:04.013 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:04.013 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:04.013 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:04.013 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:37:04.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:04.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:04.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:04.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:04.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:04.190 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:04.190 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:04.190 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:04.190 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:04.190 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:04.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:04.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:04.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:04.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:04.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:04.366 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:04.366 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:04.366 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:04.366 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:04.366 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:04.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:04.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:04.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:04.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:04.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:04.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:04.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:04.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:04.462 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.462 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.462 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:04.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:04.541 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:04.541 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:04.541 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:04.541 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:04.541 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:04.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:04.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:04.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:04.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:04.549 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.549 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.549 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:04.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:04.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:04.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:04.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:04.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:04.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:04.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:04.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:04.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:04.638 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.638 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.638 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:04.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:04.717 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:04.717 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:04.717 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:04.717 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:04.717 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:04.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:04.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:04.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:04.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:04.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:04.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2023-12-02T01:37:04.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 2 + 1 2023-12-02T01:37:04.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=3 2023-12-02T01:37:04.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 3 -gt 3 ']' 2023-12-02T01:37:04.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 384 + 0 2023-12-02T01:37:04.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=384 2023-12-02T01:37:04.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: '[' 384 -gt 0 ']' 2023-12-02T01:37:04.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: expr 3 - 2 2023-12-02T01:37:04.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:143: TEST_recover_unexpected: sleep 1 2023-12-02T01:37:05.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2023-12-02T01:37:05.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2023-12-02T01:37:05.818 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2023-12-02T01:37:05.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2023-12-02T01:37:05.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:05.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:37:05.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:37:05.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:37:05.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:05.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:37:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:37:05.910 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:05.910 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:05.910 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:05.910 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:05.910 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:05.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.0.asok 2023-12-02T01:37:05.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:05.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.0.asok dump_scrub_reservations 2023-12-02T01:37:06.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:06.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:06.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:06.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:06.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:06.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:37:06.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:37:06.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:37:06.008 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.008 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.008 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:37:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:37:06.087 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:06.087 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:06.087 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:06.087 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:06.088 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.096 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.1.asok 2023-12-02T01:37:06.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.1.asok dump_scrub_reservations 2023-12-02T01:37:06.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:06.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:06.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:06.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:06.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:06.263 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:06.263 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:06.263 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:06.263 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:06.263 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.2.asok 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.2.asok dump_scrub_reservations 2023-12-02T01:37:06.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:06.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:06.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:06.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:06.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:06.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:06.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:06.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:06.359 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:06.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:06.439 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:06.439 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:06.439 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:06.439 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:06.439 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:06.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.3.asok 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.3.asok dump_scrub_reservations 2023-12-02T01:37:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:06.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:06.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:06.612 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:06.612 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:06.612 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:06.612 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:06.612 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.4.asok 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.4.asok dump_scrub_reservations 2023-12-02T01:37:06.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:06.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:06.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:06.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:06.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2023-12-02T01:37:06.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:06.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:06.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:06.787 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2023-12-02T01:37:06.788 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_local": 0, 2023-12-02T01:37:06.788 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubs_remote": 0, 2023-12-02T01:37:06.788 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_max_scrubs": 4 2023-12-02T01:37:06.788 INFO:tasks.workunit.client.0.smithi162.stdout:} 2023-12-02T01:37:06.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2023-12-02T01:37:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2023-12-02T01:37:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2023-12-02T01:37:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2023-12-02T01:37:06.797 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-12-02T01:37:06.797 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:06.797 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:06.797 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.121566/ceph-osd.5.asok 2023-12-02T01:37:06.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2023-12-02T01:37:06.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.121566/ceph-osd.5.asok dump_scrub_reservations 2023-12-02T01:37:06.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2023-12-02T01:37:06.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2023-12-02T01:37:06.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2023-12-02T01:37:06.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2023-12-02T01:37:06.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2023-12-02T01:37:06.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 3 + 1 2023-12-02T01:37:06.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=4 2023-12-02T01:37:06.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 4 -gt 3 ']' 2023-12-02T01:37:06.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: break 2023-12-02T01:37:06.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: seq 0 5 2023-12-02T01:37:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: for i in $(seq 0 5) 2023-12-02T01:37:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:152: TEST_recover_unexpected: ceph pg dump pgs 2023-12-02T01:37:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:152: TEST_recover_unexpected: grep +scrubbing 2023-12-02T01:37:07.147 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2023-12-02T01:37:07.156 INFO:tasks.workunit.client.0.smithi162.stdout:3.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2023-12-02T01:36:57.842304+0000 42'66 45:93 [4,1,2] 4 [4,1,2] 4 0'0 2023-12-02T01:28:03.797576+0000 0'0 2023-12-02T01:28:03.797576+0000 0 0 scrubbing for 5s 50 0 2023-12-02T01:37:07.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:153: TEST_recover_unexpected: echo 'ERROR: Extra scrubs after test completion...not expected' 2023-12-02T01:37:07.157 INFO:tasks.workunit.client.0.smithi162.stdout:ERROR: Extra scrubs after test completion...not expected 2023-12-02T01:37:07.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:154: TEST_recover_unexpected: return 1 2023-12-02T01:37:07.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:46: run: return 1 2023-12-02T01:37:07.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2023-12-02T01:37:07.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 1 2023-12-02T01:37:07.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2023-12-02T01:37:07.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2023-12-02T01:37:07.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2023-12-02T01:37:07.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-12-02T01:37:07.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-12-02T01:37:07.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-12-02T01:37:07.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-12-02T01:37:07.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-12-02T01:37:07.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-12-02T01:37:07.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-12-02T01:37:07.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-12-02T01:37:07.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-12-02T01:37:07.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-12-02T01:37:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-12-02T01:37:07.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-12-02T01:37:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:37:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-12-02T01:37:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-12-02T01:37:07.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T01:37:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:37:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2023-12-02T01:37:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2023-12-02T01:37:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2023-12-02T01:37:07.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-dump/mgr.x.log td/osd-scrub-dump/mon.a.log td/osd-scrub-dump/osd.0.log td/osd-scrub-dump/osd.1.log td/osd-scrub-dump/osd.2.log td/osd-scrub-dump/osd.3.log td/osd-scrub-dump/osd.4.log td/osd-scrub-dump/osd.5.log /home/ubuntu/cephtest/archive/log 2023-12-02T01:37:07.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2023-12-02T01:37:07.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-12-02T01:37:07.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-12-02T01:37:07.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.121566 2023-12-02T01:37:07.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.121566 2023-12-02T01:37:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T01:37:07.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-12-02T01:37:07.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-12-02T01:37:07.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2023-12-02T01:37:07.294 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2023-12-02T01:37:07.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-02T01:37:07.608 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-dump.sh) on smithi162 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh' 2023-12-02T01:37:07.697 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3ff8243e1eeb46f88599885638f00798 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-dump.sh) on smithi162 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh' 2023-12-02T01:37:07.701 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-02T01:37:07.711 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-02T01:37:07.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-02T01:37:07.768 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-02T01:37:07.777 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-02T01:37:07.777 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T01:37:07.798 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2023-12-02T01:37:07.800 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T01:37:07.800 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2023-12-02T01:37:07.801 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +956us[ +956us] +/- 76ms 2023-12-02T01:37:07.801 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -444us[ -444us] +/- 52ms 2023-12-02T01:37:07.801 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -280us[ -280us] +/- 79ms 2023-12-02T01:37:07.801 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T01:37:07.802 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-02T01:37:07.811 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-02T01:37:07.811 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-02T01:37:07.819 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-02T01:37:07.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T01:37:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T01:37:07.970 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-02T01:37:07.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-02T01:37:07.989 INFO:teuthology.task.internal:Duration was 1799.157825 seconds 2023-12-02T01:37:07.989 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-02T01:37:07.998 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-02T01:37:07.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-02T01:37:08.053 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T01:37:08.532 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-02T01:37:08.533 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2023-12-02T01:37:08.533 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-02T01:37:08.599 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-02T01:37:08.600 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-02T01:37:08.669 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-02T01:37:08.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-02T01:37:08.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-02T01:37:08.807 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-02T01:37:08.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-02T01:37:08.835 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-02T01:37:08.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:37:08.901 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2023-12-02T01:37:08.924 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-02T01:37:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T01:37:08.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-02T01:37:08.987 INFO:teuthology.task.internal:Transferring archived files... 2023-12-02T01:37:08.989 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi/7474671/remote/smithi162 2023-12-02T01:37:08.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-02T01:37:34.033 INFO:teuthology.task.internal:Removing archive directory... 2023-12-02T01:37:34.034 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-02T01:37:34.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-02T01:37:34.185 INFO:teuthology.task.internal:Not uploading archives. 2023-12-02T01:37:34.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-02T01:37:34.194 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-02T01:37:34.194 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-02T01:37:34.209 INFO:teuthology.orchestra.run.smithi162.stdout: 265305 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 2 01:37 /home/ubuntu/cephtest 2023-12-02T01:37:34.209 INFO:teuthology.orchestra.run.smithi162.stdout: 276831 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 2 01:15 /home/ubuntu/cephtest/mnt.0 2023-12-02T01:37:34.209 INFO:teuthology.orchestra.run.smithi162.stdout: 276832 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 2 01:26 /home/ubuntu/cephtest/mnt.0/client.0 2023-12-02T01:37:34.209 INFO:teuthology.orchestra.run.smithi162.stdout: 393217 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 2 01:35 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-02T01:37:34.210 INFO:teuthology.orchestra.run.smithi162.stdout: 393218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 01:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2023-12-02T01:37:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T01:37:34.211 INFO:teuthology.orchestra.run.smithi162.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-02T01:37:34.211 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-02T01:37:34.216 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-02T01:37:34.226 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-02T01:37:34.235 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi162 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-02T01:37:34.236 INFO:teuthology.nuke:Checking targets against current locks 2023-12-02T01:37:34.262 DEBUG:teuthology.nuke:shortname: smithi162 2023-12-02T01:37:34.262 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T01:37:34.282 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_17:58:18-rados-main-distro-default-smithi/7474671', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 00:58:48.787937', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPR7i45QAtTsn88VNEd9XXlKC4pSCTBbv/WfCIDn1qnRaWhTGoTDrSoN5HiZ/jk3/ndyrs7Jr0VgLPJWRLqbCSM='} 2023-12-02T01:37:34.334 INFO:teuthology.orchestra.console.smithi162:Power off 2023-12-02T01:37:34.334 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-02T01:37:34.361 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2023-12-02T01:37:34.361 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T01:37:34.385 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2023-12-02T01:37:38.387 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T01:37:38.497 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2023-12-02T01:37:42.502 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T01:37:42.612 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2023-12-02T01:37:46.614 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T01:37:46.726 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2023-12-02T01:37:46.827 INFO:teuthology.orchestra.console.smithi162:Power off completed 2023-12-02T01:37:46.932 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 1799.1578245162964 failure_reason: 'Command failed (workunit test scrub/osd-scrub-dump.sh) on smithi162 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3ff8243e1eeb46f88599885638f00798 status: fail success: false 2023-12-02T01:37:46.932 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-02T01:37:47.009 INFO:teuthology.run:FAIL